Jul 06 19:09:34.401010 ip-10-0-142-236 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:09:36.241152 ip-10-0-142-236 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:09:36.241152 ip-10-0-142-236 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:09:36.241152 ip-10-0-142-236 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:09:36.241152 ip-10-0-142-236 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:09:36.241152 ip-10-0-142-236 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:09:36.241887 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.241154 2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247780 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247801 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247805 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247808 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247811 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247814 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247818 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247821 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247825 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247828 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247831 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247833 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:09:36.247821 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247836 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247840 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247843 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247846 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247849 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247852 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247855 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247861 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247864 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247866 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247869 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247872 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247875 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247879 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247884 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247887 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247890 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247893 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247896 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247899 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:09:36.248233 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247902 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247905 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247908 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247911 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247914 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247916 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247919 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247922 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247925 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247927 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247930 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247932 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247935 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247938 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247942 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247945 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247948 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247951 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247955 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247957 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:09:36.248710 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247960 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247963 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247966 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247968 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247971 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247973 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247976 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247979 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247981 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247984 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247987 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247991 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247995 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.247998 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248001 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248004 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248007 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248010 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248013 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:09:36.249239 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248015 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248017 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248020 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248023 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248025 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248028 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248033 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248038 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248042 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248045 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248048 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248050 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248053 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248056 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248059 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248061 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248064 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248067 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248069 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:09:36.249712 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248072 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248075 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248078 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248080 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248083 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248086 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248089 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248091 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248094 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248924 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248932 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248936 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248940 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248943 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248946 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248948 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248951 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248954 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248957 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248960 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:09:36.250286 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248963 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248966 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248969 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248972 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248975 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248978 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248982 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248985 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248987 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248992 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248996 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.248999 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249001 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249005 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249007 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249010 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249012 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249015 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249018 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249020 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:09:36.250784 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249023 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249026 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249029 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249031 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249034 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249037 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249040 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249043 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249046 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249049 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249051 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249054 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249057 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249060 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249065 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249068 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249071 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249075 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249078 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:09:36.251316 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249081 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249084 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249087 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249090 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249092 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249095 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249097 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249100 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249103 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249105 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249108 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249111 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249113 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249116 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249119 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249122 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249124 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249127 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249130 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249133 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:09:36.251794 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249135 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249138 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249141 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249145 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249150 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249153 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249156 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249159 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249163 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249166 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249169 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249171 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249174 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249177 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249180 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249183 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249186 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249189 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249192 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:09:36.252331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249194 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249197 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249199 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249202 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249205 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249207 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249210 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249212 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249215 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249218 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249663 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249671 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249675 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249678 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249681 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249683 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249686 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249689 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249692 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249695 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:09:36.252801 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249697 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249700 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249703 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249706 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249709 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249712 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249715 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249718 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249721 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249724 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249726 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249729 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249732 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249734 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249737 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249740 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249742 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249745 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249748 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249750 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:09:36.253362 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249753 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249757 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249759 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249762 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249765 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249768 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249771 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249773 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249776 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249779 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249782 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249784 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249787 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249790 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249793 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249796 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249799 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249801 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249820 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249825 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:09:36.253961 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249828 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249830 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249833 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249836 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249839 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249842 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249844 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249847 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249850 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249852 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249855 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249858 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249860 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249864 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249867 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249870 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249873 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249876 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249878 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:09:36.254465 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249881 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249884 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249886 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249889 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249892 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249895 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249899 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249902 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249905 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249909 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249912 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249915 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249918 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249922 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249924 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249927 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249930 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249932 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249935 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:09:36.254984 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249937 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249940 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249942 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249945 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249948 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249950 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249952 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249956 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249960 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249964 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.249967 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250057 2554 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250064 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250072 2554 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250079 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250086 2554 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250091 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250098 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250104 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250108 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250112 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:09:36.255459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250115 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250118 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250121 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250125 2554 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250128 2554 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250131 2554 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250134 2554 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250137 2554 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250142 2554 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250145 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250148 2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250151 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250154 2554 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250159 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250162 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250165 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250168 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250171 2554 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250179 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250182 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250186 2554 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250189 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250194 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250197 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250200 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:09:36.256000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250203 2554 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250206 2554 flags.go:64] FLAG: --enable-server="true" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250209 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250214 2554 flags.go:64] FLAG: --event-burst="100" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250217 2554 flags.go:64] FLAG: --event-qps="50" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250220 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250223 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250226 2554 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250230 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250233 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250236 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250239 2554 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250242 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250245 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250249 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250252 2554 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250255 2554 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250258 2554 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250261 2554 flags.go:64] FLAG: --feature-gates="" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250265 2554 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250268 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250271 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250274 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250277 2554 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250280 2554 flags.go:64] FLAG: --help="false" Jul 06 19:09:36.256615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250288 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250292 2554 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250295 2554 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250299 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250302 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250306 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250309 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250312 2554 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250315 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250318 2554 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250321 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250324 2554 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250327 2554 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250330 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250333 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250336 2554 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250339 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250342 2554 flags.go:64] FLAG: --lock-file="" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250345 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250348 2554 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250351 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250357 2554 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250361 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250364 2554 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:09:36.257311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250367 2554 flags.go:64] FLAG: --logging-format="text" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250370 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250374 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250376 2554 flags.go:64] FLAG: --manifest-url="" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250379 2554 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250384 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250387 2554 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250391 2554 flags.go:64] FLAG: --max-pods="110" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250396 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250400 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250403 2554 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250406 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250409 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250412 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250415 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250426 2554 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250429 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250432 2554 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250435 2554 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250438 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250444 2554 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250447 2554 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250450 2554 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250453 2554 flags.go:64] FLAG: --port="10250" Jul 06 19:09:36.257971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250457 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250460 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01615e2690f706c91" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250463 2554 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250466 2554 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250469 2554 flags.go:64] FLAG: --register-node="true" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250472 2554 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250476 2554 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250479 2554 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250483 2554 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250486 2554 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250490 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250493 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250496 2554 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250499 2554 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250502 2554 flags.go:64] FLAG: --runonce="false" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250505 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250509 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250514 2554 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250517 2554 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250521 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250524 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250527 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250531 2554 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250534 2554 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250537 2554 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:09:36.258576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250540 2554 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250543 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250546 2554 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250549 2554 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250552 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250558 2554 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250561 2554 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250564 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250567 2554 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250570 2554 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250573 2554 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250576 2554 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250579 2554 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250582 2554 flags.go:64] FLAG: --v="2" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250586 2554 flags.go:64] FLAG: --version="false" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250591 2554 flags.go:64] FLAG: --vmodule="" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250595 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.250598 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250722 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250725 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250728 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250731 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250734 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:09:36.259307 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250738 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250743 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250746 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250749 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250752 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250755 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250758 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250761 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250763 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250766 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250769 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250772 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250774 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250777 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250780 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250783 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250786 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250789 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250792 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250795 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:09:36.259906 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250798 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250800 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250817 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250820 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250823 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250827 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250830 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250832 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250835 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250838 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250841 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250843 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250849 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250854 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250857 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250860 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250864 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250868 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250871 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:09:36.260409 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250874 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250877 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250880 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250883 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250886 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250889 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250891 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250894 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250897 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250900 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250902 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250905 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250908 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250911 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250913 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250916 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250919 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250922 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250924 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250928 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:09:36.260904 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250931 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250934 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250937 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250940 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250942 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250946 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250951 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250954 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250956 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250959 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250962 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250964 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250967 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250970 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250973 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250975 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250978 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250981 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250984 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:09:36.261397 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250986 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250989 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250992 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250995 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.250997 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251000 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251003 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251005 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251009 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251013 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251017 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251020 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251023 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251026 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251029 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:09:36.261942 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.251032 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.251940 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.259570 2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.259588 2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.259604 2554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.259614 2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259673 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259679 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259683 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259686 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259689 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259693 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259697 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:09:36.262327 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259700 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259704 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259709 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259712 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259715 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259719 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259723 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259727 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259729 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259732 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259735 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259738 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259741 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259743 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259746 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259749 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259752 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259754 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259757 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:09:36.262695 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259760 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259762 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259765 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259769 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259773 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259776 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259778 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259781 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259784 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259787 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259789 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259792 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259795 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259798 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259800 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259803 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259824 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259828 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259830 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259833 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:09:36.263204 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259836 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259839 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259842 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259844 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259847 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259850 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259853 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259856 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259858 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259861 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259864 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259866 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259869 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259872 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259874 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259878 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259882 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259885 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259888 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259890 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259893 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:09:36.263714 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259896 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259899 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259901 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259905 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259907 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259910 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259913 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259916 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259919 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259922 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259925 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259928 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259931 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259933 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259936 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259938 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259941 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259944 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259946 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259949 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:09:36.264254 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259952 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259955 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259957 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259960 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259962 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259965 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259969 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259972 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259975 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259977 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259980 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.259983 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.259988 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260107 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:09:36.264740 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260111 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260116 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260120 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260123 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260126 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260129 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260132 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260135 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260138 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260140 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260143 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260146 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260149 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260151 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260154 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260157 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260160 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260162 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260165 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:09:36.265105 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260168 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260170 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260173 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260176 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260178 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260183 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260186 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260189 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260192 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260195 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260198 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260200 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260203 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260205 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260209 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260211 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260214 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260217 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260219 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:09:36.265583 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260222 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260225 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260227 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260230 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260233 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260235 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260238 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260241 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260244 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260246 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260249 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260252 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260254 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260258 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260262 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260265 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260268 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260271 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260275 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:09:36.266132 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260280 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260284 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260287 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260290 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260293 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260296 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260299 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260301 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260304 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260307 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260310 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260312 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260315 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260318 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260320 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260323 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260325 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260328 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260331 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260334 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:09:36.266603 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260336 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260338 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260341 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260344 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260346 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260349 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260351 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260354 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260357 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260359 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260362 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260365 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260368 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260370 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260373 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260376 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260379 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260381 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260384 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260387 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:09:36.267115 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:36.260389 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:09:36.267616 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.260394 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:09:36.267616 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.260513 2554 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:09:36.267616 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.263227 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:09:36.267616 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.264896 2554 server.go:1019] "Starting client certificate rotation" Jul 06 19:09:36.267616 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.265003 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:09:36.267616 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.265042 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:09:36.267616 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.267518 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:09:36.269010 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.268992 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:09:36.282587 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.282562 2554 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:09:36.285236 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.285220 2554 log.go:25] "Validated CRI v1 image API" Jul 06 19:09:36.286167 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.286150 2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:09:36.291434 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.291411 2554 fs.go:135] Filesystem UUIDs: map[0af0e4da-41a9-4be1-8b10-837bfe55eeda:/dev/nvme0n1p4 3cf56bf6-5269-4936-a2af-df9d62202916:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 06 19:09:36.291511 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.291432 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:09:36.296049 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.296032 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:09:36.296818 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.296699 2554 manager.go:216] Machine: {Timestamp:2026-07-06 19:09:36.295341494 +0000 UTC m=+1.778565061 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199885 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27a5cfa00afca718951b3c6fe675b1 SystemUUID:ec27a5cf-a00a-fca7-1895-1b3c6fe675b1 BootID:2511ec18-9aac-4b04-9bec-12db327af84b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8e:f9:ba:1d:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8e:f9:ba:1d:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:64:a3:91:79:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:09:36.296879 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.296825 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:09:36.296972 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.296959 2554 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:09:36.300389 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.300364 2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:09:36.300527 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.300392 2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:09:36.301239 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.301229 2554 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:09:36.301277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.301242 2554 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:09:36.301277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.301250 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:09:36.302078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.302066 2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:09:36.304142 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.304133 2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:09:36.308025 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.308013 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:09:36.308144 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.308134 2554 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:09:36.308180 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.308167 2554 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:09:36.308210 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.308180 2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:09:36.308210 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.308202 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:09:36.308268 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.308217 2554 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:09:36.308268 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.308226 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:09:36.308626 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.308614 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:09:36.308664 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.308633 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:09:36.311232 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.311215 2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:09:36.313519 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.313502 2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:09:36.313619 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.313543 2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:09:36.315645 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315631 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:09:36.315645 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315640 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pr8np" Jul 06 19:09:36.315771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315652 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:09:36.315771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315662 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:09:36.315771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315671 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:09:36.315771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315685 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:09:36.315771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315695 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:09:36.315771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315724 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:09:36.315771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315733 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:09:36.315771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315745 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:09:36.315771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315754 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:09:36.315771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315773 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:09:36.316078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.315782 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:09:36.316647 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.316636 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:09:36.316697 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.316650 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:09:36.317540 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.317507 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:09:36.317619 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.317561 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:09:36.319706 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.319689 2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:09:36.320141 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.320129 2554 server.go:1268] "Started kubelet" Jul 06 19:09:36.320428 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.320241 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:09:36.320428 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.320245 2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:09:36.320428 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.320327 2554 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:09:36.321089 ip-10-0-142-236 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:09:36.322447 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.322434 2554 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:09:36.323971 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.323951 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pr8np" Jul 06 19:09:36.325614 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.325598 2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:09:36.326466 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.326446 2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:09:36.332284 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.331293 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-236.ec2.internal.18bfc8ff4e6eb77f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-236.ec2.internal,UID:ip-10-0-142-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-236.ec2.internal,},FirstTimestamp:2026-07-06 19:09:36.320100223 +0000 UTC m=+1.803323790,LastTimestamp:2026-07-06 19:09:36.320100223 +0000 UTC m=+1.803323790,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-236.ec2.internal,}" Jul 06 19:09:36.335598 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.335576 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:09:36.335694 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.335598 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:09:36.335694 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.335685 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:36.336493 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.336369 2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:09:36.338001 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.337602 2554 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:09:36.338001 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.337701 2554 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:09:36.338001 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.337885 2554 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:09:36.338001 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.337894 2554 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:09:36.340653 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.340627 2554 factory.go:153] Registering CRI-O factory Jul 06 19:09:36.340653 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.340652 2554 factory.go:223] Registration of the crio container factory successfully Jul 06 19:09:36.340822 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.340711 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:09:36.340822 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.340720 2554 factory.go:55] Registering systemd factory Jul 06 19:09:36.340822 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.340730 2554 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:09:36.340822 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.340754 2554 factory.go:103] Registering Raw factory Jul 06 19:09:36.340822 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.340803 2554 manager.go:1196] Started watching for new ooms in manager Jul 06 19:09:36.342469 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.342447 2554 manager.go:349] Starting recovery of all containers Jul 06 19:09:36.349722 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.349700 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:09:36.352105 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.352091 2554 manager.go:354] Recovery completed Jul 06 19:09:36.356549 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.356534 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:09:36.356629 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.356565 2554 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-236.ec2.internal\" not found" node="ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.358691 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.358676 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:09:36.358784 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.358709 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:09:36.358784 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.358719 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:09:36.359382 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.359366 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:09:36.359382 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.359380 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:09:36.359483 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.359384 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:09:36.359483 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.359397 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:09:36.360494 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.360480 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:09:36.360541 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.360505 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:09:36.360541 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.360516 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:09:36.360797 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.360786 2554 policy_none.go:49] "None policy: Start" Jul 06 19:09:36.360844 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.360801 2554 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:09:36.360844 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.360811 2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:09:36.361948 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.361937 2554 policy_none.go:47] "Start" Jul 06 19:09:36.397604 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.397454 2554 manager.go:337] "Starting Device Plugin manager" Jul 06 19:09:36.397720 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.397631 2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:09:36.397720 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.397643 2554 server.go:87] "Starting device plugin registration server" Jul 06 19:09:36.397891 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.397878 2554 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:09:36.397945 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.397892 2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:09:36.398019 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.397998 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:09:36.398111 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.398097 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:09:36.398111 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.398110 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:09:36.398679 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.398657 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:09:36.398778 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.398718 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:36.498940 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.498870 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:09:36.500278 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.499975 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:09:36.500278 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.500013 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:09:36.500278 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.500028 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:09:36.500278 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.500061 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.520106 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.520084 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.520186 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.520108 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-236.ec2.internal\": node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:36.522154 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.522117 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:09:36.523380 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.523356 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:09:36.523479 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.523408 2554 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:09:36.523479 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.523437 2554 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:09:36.523565 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.523481 2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:09:36.526839 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.526817 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:09:36.587593 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.587562 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:36.626074 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.626035 2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"] Jul 06 19:09:36.626203 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.626119 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:09:36.628033 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.628012 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:09:36.628139 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.628046 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:09:36.628139 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.628057 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:09:36.629688 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.629675 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:09:36.629866 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.629850 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.629914 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.629882 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:09:36.630525 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.630506 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:09:36.630614 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.630540 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:09:36.630614 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.630554 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:09:36.630614 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.630514 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:09:36.630763 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.630617 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:09:36.630763 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.630633 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:09:36.632071 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.632052 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.632167 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.632085 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:09:36.632790 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.632773 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:09:36.632901 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.632800 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:09:36.632901 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.632833 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:09:36.639288 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.639267 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/69103f5be0b7c8797e5968b28b95762b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"69103f5be0b7c8797e5968b28b95762b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.639380 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.639294 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6749d73f2a1c44df670398dd9fe19471-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"6749d73f2a1c44df670398dd9fe19471\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.639380 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.639314 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6749d73f2a1c44df670398dd9fe19471-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"6749d73f2a1c44df670398dd9fe19471\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.656572 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.656551 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-236.ec2.internal\" not found" node="ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.661383 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.661365 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-236.ec2.internal\" not found" node="ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.688280 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.688250 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:36.739432 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.739406 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6749d73f2a1c44df670398dd9fe19471-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"6749d73f2a1c44df670398dd9fe19471\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.739569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.739441 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6749d73f2a1c44df670398dd9fe19471-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"6749d73f2a1c44df670398dd9fe19471\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.739569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.739461 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/69103f5be0b7c8797e5968b28b95762b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"69103f5be0b7c8797e5968b28b95762b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.739569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.739503 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/69103f5be0b7c8797e5968b28b95762b-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"69103f5be0b7c8797e5968b28b95762b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.739569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.739507 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6749d73f2a1c44df670398dd9fe19471-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"6749d73f2a1c44df670398dd9fe19471\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.739569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.739513 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6749d73f2a1c44df670398dd9fe19471-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"6749d73f2a1c44df670398dd9fe19471\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.789032 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.788943 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:36.889788 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.889706 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:36.961105 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.961072 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.966136 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:36.965783 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Jul 06 19:09:36.990129 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:36.990102 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:37.093206 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:37.093101 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:37.197281 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:37.197232 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:37.264832 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.264804 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:09:37.265280 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.264961 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:09:37.265280 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.265000 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:09:37.299775 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:37.299710 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:37.330846 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.330788 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:04:36 +0000 UTC" deadline="2028-04-13 18:50:32.52904957 +0000 UTC" Jul 06 19:09:37.330846 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.330838 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15527h40m55.198214829s" Jul 06 19:09:37.400413 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:37.400312 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:37.501165 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:37.501129 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Jul 06 19:09:37.517416 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.517385 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:09:37.536102 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.536073 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Jul 06 19:09:37.547850 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.547825 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:09:37.549521 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.549504 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Jul 06 19:09:37.559876 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.559850 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:09:37.619758 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:37.619718 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69103f5be0b7c8797e5968b28b95762b.slice/crio-4d6f37d6b8e3f0ad90a281ae781af4cf5015793b96d7e2221ee4b806360611d3 WatchSource:0}: Error finding container 4d6f37d6b8e3f0ad90a281ae781af4cf5015793b96d7e2221ee4b806360611d3: Status 404 returned error can't find the container with id 4d6f37d6b8e3f0ad90a281ae781af4cf5015793b96d7e2221ee4b806360611d3 Jul 06 19:09:37.620169 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:37.620132 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6749d73f2a1c44df670398dd9fe19471.slice/crio-5b332baa6d236ab83ea3d88883b19f989256202f108c1587ada53d8bb11b75d7 WatchSource:0}: Error finding container 5b332baa6d236ab83ea3d88883b19f989256202f108c1587ada53d8bb11b75d7: Status 404 returned error can't find the container with id 5b332baa6d236ab83ea3d88883b19f989256202f108c1587ada53d8bb11b75d7 Jul 06 19:09:37.624647 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.624632 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:09:37.866106 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:37.866022 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:09:38.309240 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.309144 2554 apiserver.go:52] "Watching apiserver" Jul 06 19:09:38.317118 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.316905 2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:09:38.317408 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.317369 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-czg45","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal","openshift-multus/multus-additional-cni-plugins-xlh9g","openshift-multus/network-metrics-daemon-hlrkz","openshift-network-diagnostics/network-check-target-g5gwx","openshift-network-operator/iptables-alerter-8j7nt","kube-system/konnectivity-agent-x2dvn","kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal","openshift-dns/node-resolver-56v7k","openshift-image-registry/node-ca-xqhsh","openshift-multus/multus-kj27t","openshift-ovn-kubernetes/ovnkube-node-bfz6c","kube-system/global-pull-secret-syncer-ncc6d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh"] Jul 06 19:09:38.319353 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.319329 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.320938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.320893 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:09:38.322577 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.322353 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:09:38.322577 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.322419 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:09:38.322577 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.322477 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tw488\"" Jul 06 19:09:38.322873 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.322661 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:09:38.323588 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.323534 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:09:38.323588 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.323557 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hk2jv\"" Jul 06 19:09:38.324192 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.324003 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:09:38.325809 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.324800 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.327330 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.326648 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.329001 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.327548 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.329001 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.328901 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.332416 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.332323 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4t98n\"" Jul 06 19:09:38.332637 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.332573 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.332837 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.332853 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.333356 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.333591 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dpwdb\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.334264 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.334304 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.334328 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8pksp\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.334867 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.335235 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cb7hv\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.335431 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.334869 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.335719 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.335967 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.336168 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.336553 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:09:38.337277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.336897 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:09:38.338505 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.338486 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:38.338591 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.338560 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:38.339840 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.339814 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.341546 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.341502 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:38.341633 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.341602 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:38.342625 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.342603 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:09:38.342880 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.342860 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g9k42\"" Jul 06 19:09:38.343257 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.343238 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:09:38.345036 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.345015 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.345918 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.345559 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.346447 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.346354 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:38.346535 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.346472 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:38.347636 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.347603 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:09:38.347733 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.347673 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fpcgn\"" Jul 06 19:09:38.348062 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.348041 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:09:38.349507 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.349491 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vl4lx\"" Jul 06 19:09:38.349688 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.349669 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:09:38.349869 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.349618 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:09:38.350075 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350058 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:09:38.350306 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350282 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:09:38.350389 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350364 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:09:38.350507 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350486 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4vg\" (UniqueName: \"kubernetes.io/projected/598c2c71-e257-4445-8f36-710b469c366f-kube-api-access-8d4vg\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.350577 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350523 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-sysctl-conf\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.350577 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350548 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-sys\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.350577 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350572 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-system-cni-dir\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.350727 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350596 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-cnibin\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.350727 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350621 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/598c2c71-e257-4445-8f36-710b469c366f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.350727 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350647 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-sys-fs\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.350727 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350671 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcc9b3b-8cfd-4704-884c-36d40629bf51-tmp\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.350727 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350695 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8bb\" (UniqueName: \"kubernetes.io/projected/eaebfdaf-a012-490a-8e17-b8346cbf6748-kube-api-access-bx8bb\") pod \"node-ca-xqhsh\" (UID: \"eaebfdaf-a012-490a-8e17-b8346cbf6748\") " pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.351002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350729 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd8720ca-7ec0-425b-9b19-2afaf1699d4d-konnectivity-ca\") pod \"konnectivity-agent-x2dvn\" (UID: \"dd8720ca-7ec0-425b-9b19-2afaf1699d4d\") " pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:09:38.351002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350753 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-os-release\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.351002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350777 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-modprobe-d\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.351002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350800 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-sysctl-d\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.351002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350843 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxdr\" (UniqueName: \"kubernetes.io/projected/444b4377-ba44-4941-9f99-1136ba12a661-kube-api-access-cvxdr\") pod \"iptables-alerter-8j7nt\" (UID: \"444b4377-ba44-4941-9f99-1136ba12a661\") " pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.351002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350905 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/423a3192-16a1-4e66-94de-9d88c359469c-kubelet-config\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:38.351002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350932 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eaebfdaf-a012-490a-8e17-b8346cbf6748-serviceca\") pod \"node-ca-xqhsh\" (UID: \"eaebfdaf-a012-490a-8e17-b8346cbf6748\") " pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.351002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350959 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd8720ca-7ec0-425b-9b19-2afaf1699d4d-agent-certs\") pod \"konnectivity-agent-x2dvn\" (UID: \"dd8720ca-7ec0-425b-9b19-2afaf1699d4d\") " pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:09:38.351002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.350996 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/598c2c71-e257-4445-8f36-710b469c366f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351026 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-systemd\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351069 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-lib-modules\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351107 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-host\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351148 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/423a3192-16a1-4e66-94de-9d88c359469c-dbus\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351166 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351172 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eaebfdaf-a012-490a-8e17-b8346cbf6748-host\") pod \"node-ca-xqhsh\" (UID: \"eaebfdaf-a012-490a-8e17-b8346cbf6748\") " pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351200 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-etc-selinux\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351247 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-sysconfig\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351283 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-run\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351398 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-var-lib-kubelet\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.351427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351428 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/444b4377-ba44-4941-9f99-1136ba12a661-iptables-alerter-script\") pod \"iptables-alerter-8j7nt\" (UID: \"444b4377-ba44-4941-9f99-1136ba12a661\") " pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351458 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-socket-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351494 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-device-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351504 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351519 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2844\" (UniqueName: \"kubernetes.io/projected/aef3e33d-1cbf-41e5-97a7-d821055fc115-kube-api-access-d2844\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351545 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-kubernetes\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351570 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351599 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/598c2c71-e257-4445-8f36-710b469c366f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351653 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351727 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-tuned\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351752 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351795 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-registration-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351826 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrpl\" (UniqueName: \"kubernetes.io/projected/3bcc9b3b-8cfd-4704-884c-36d40629bf51-kube-api-access-6qrpl\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.352030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.351853 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/444b4377-ba44-4941-9f99-1136ba12a661-host-slash\") pod \"iptables-alerter-8j7nt\" (UID: \"444b4377-ba44-4941-9f99-1136ba12a661\") " pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.372521 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.371204 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m5z2v" Jul 06 19:09:38.380854 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.380242 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m5z2v" Jul 06 19:09:38.401176 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.401156 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:09:38.437575 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.437542 2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:09:38.452370 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452340 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-sysctl-conf\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.452514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452380 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-sys\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.452514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452411 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-os-release\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.452514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452436 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-cni-netd\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.452514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452468 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8724e53-459c-4708-aa69-9ca66d0f90ab-ovnkube-config\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.452514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452501 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-sys\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452515 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-sys-fs\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452543 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-sysctl-conf\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452555 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcc9b3b-8cfd-4704-884c-36d40629bf51-tmp\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452583 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-sys-fs\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452589 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-var-lib-cni-bin\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452617 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-daemon-config\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452643 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452671 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-run-k8s-cni-cncf-io\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452711 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd8720ca-7ec0-425b-9b19-2afaf1699d4d-konnectivity-ca\") pod \"konnectivity-agent-x2dvn\" (UID: \"dd8720ca-7ec0-425b-9b19-2afaf1699d4d\") " pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452752 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-modprobe-d\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452777 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-sysctl-d\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452804 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/423a3192-16a1-4e66-94de-9d88c359469c-kubelet-config\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:38.452856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452828 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eaebfdaf-a012-490a-8e17-b8346cbf6748-serviceca\") pod \"node-ca-xqhsh\" (UID: \"eaebfdaf-a012-490a-8e17-b8346cbf6748\") " pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452892 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-modprobe-d\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452932 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452963 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/423a3192-16a1-4e66-94de-9d88c359469c-kubelet-config\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.452990 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-sysctl-d\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453033 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8724e53-459c-4708-aa69-9ca66d0f90ab-ovn-node-metrics-cert\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453060 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1c2a219-c8c6-4937-952e-630b3eb7de5a-hosts-file\") pod \"node-resolver-56v7k\" (UID: \"c1c2a219-c8c6-4937-952e-630b3eb7de5a\") " pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453085 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-socket-dir-parent\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453116 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/598c2c71-e257-4445-8f36-710b469c366f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453150 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eaebfdaf-a012-490a-8e17-b8346cbf6748-host\") pod \"node-ca-xqhsh\" (UID: \"eaebfdaf-a012-490a-8e17-b8346cbf6748\") " pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453198 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8724e53-459c-4708-aa69-9ca66d0f90ab-ovnkube-script-lib\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453227 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-etc-selinux\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453252 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-run\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453279 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/444b4377-ba44-4941-9f99-1136ba12a661-iptables-alerter-script\") pod \"iptables-alerter-8j7nt\" (UID: \"444b4377-ba44-4941-9f99-1136ba12a661\") " pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453306 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j6h8\" (UniqueName: \"kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8\") pod \"network-check-target-g5gwx\" (UID: \"9b975cfc-6a57-4352-8833-61170b28d14b\") " pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453312 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eaebfdaf-a012-490a-8e17-b8346cbf6748-host\") pod \"node-ca-xqhsh\" (UID: \"eaebfdaf-a012-490a-8e17-b8346cbf6748\") " pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453306 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd8720ca-7ec0-425b-9b19-2afaf1699d4d-konnectivity-ca\") pod \"konnectivity-agent-x2dvn\" (UID: \"dd8720ca-7ec0-425b-9b19-2afaf1699d4d\") " pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:09:38.453478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453331 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-conf-dir\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453359 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-systemd-units\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453380 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/eaebfdaf-a012-490a-8e17-b8346cbf6748-serviceca\") pod \"node-ca-xqhsh\" (UID: \"eaebfdaf-a012-490a-8e17-b8346cbf6748\") " pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453438 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-run\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453444 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-etc-selinux\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453475 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-log-socket\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453506 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8724e53-459c-4708-aa69-9ca66d0f90ab-env-overrides\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453535 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-socket-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453565 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-device-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453588 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-kubernetes\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453611 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453639 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-cni-dir\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453642 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-device-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453664 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-var-lib-cni-multus\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453691 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453696 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-socket-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453719 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/598c2c71-e257-4445-8f36-710b469c366f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.454393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453753 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-kubernetes\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.453754 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453782 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453808 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b89320cc-3dc2-406c-aa36-62aa5e8f377b-cni-binary-copy\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453690 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/598c2c71-e257-4445-8f36-710b469c366f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453845 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-slash\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.453862 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret podName:423a3192-16a1-4e66-94de-9d88c359469c nodeName:}" failed. No retries permitted until 2026-07-06 19:09:38.953839784 +0000 UTC m=+4.437063343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret") pod "global-pull-secret-syncer-ncc6d" (UID: "423a3192-16a1-4e66-94de-9d88c359469c") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453929 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/444b4377-ba44-4941-9f99-1136ba12a661-iptables-alerter-script\") pod \"iptables-alerter-8j7nt\" (UID: \"444b4377-ba44-4941-9f99-1136ba12a661\") " pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.453997 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454008 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454024 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-registration-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454050 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrpl\" (UniqueName: \"kubernetes.io/projected/3bcc9b3b-8cfd-4704-884c-36d40629bf51-kube-api-access-6qrpl\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454068 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454091 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aef3e33d-1cbf-41e5-97a7-d821055fc115-registration-dir\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454109 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-cnibin\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454137 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-run-multus-certs\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.455078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454166 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4vg\" (UniqueName: \"kubernetes.io/projected/598c2c71-e257-4445-8f36-710b469c366f-kube-api-access-8d4vg\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454194 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pspnq\" (UniqueName: \"kubernetes.io/projected/c1c2a219-c8c6-4937-952e-630b3eb7de5a-kube-api-access-pspnq\") pod \"node-resolver-56v7k\" (UID: \"c1c2a219-c8c6-4937-952e-630b3eb7de5a\") " pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454219 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-etc-openvswitch\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454240 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbcj\" (UniqueName: \"kubernetes.io/projected/ee50f9b8-fc16-4951-a486-88d460005a0c-kube-api-access-9mbcj\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454266 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-system-cni-dir\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454287 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/598c2c71-e257-4445-8f36-710b469c366f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454306 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-cnibin\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454343 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-system-cni-dir\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454382 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-cnibin\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454388 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/598c2c71-e257-4445-8f36-710b469c366f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454433 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8bb\" (UniqueName: \"kubernetes.io/projected/eaebfdaf-a012-490a-8e17-b8346cbf6748-kube-api-access-bx8bb\") pod \"node-ca-xqhsh\" (UID: \"eaebfdaf-a012-490a-8e17-b8346cbf6748\") " pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454526 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-run-systemd\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454552 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-os-release\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454574 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxdr\" (UniqueName: \"kubernetes.io/projected/444b4377-ba44-4941-9f99-1136ba12a661-kube-api-access-cvxdr\") pod \"iptables-alerter-8j7nt\" (UID: \"444b4377-ba44-4941-9f99-1136ba12a661\") " pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454597 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454618 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c2a219-c8c6-4937-952e-630b3eb7de5a-tmp-dir\") pod \"node-resolver-56v7k\" (UID: \"c1c2a219-c8c6-4937-952e-630b3eb7de5a\") " pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.455938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454641 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd8720ca-7ec0-425b-9b19-2afaf1699d4d-agent-certs\") pod \"konnectivity-agent-x2dvn\" (UID: \"dd8720ca-7ec0-425b-9b19-2afaf1699d4d\") " pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454649 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/598c2c71-e257-4445-8f36-710b469c366f-os-release\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454662 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-systemd\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454682 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-lib-modules\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454702 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-host\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454721 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/423a3192-16a1-4e66-94de-9d88c359469c-dbus\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454744 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-hostroot\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454785 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-sysconfig\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454809 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-var-lib-kubelet\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.455104 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-kubelet\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.455411 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-host\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.455655 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-run-openvswitch\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.455706 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g9qf\" (UniqueName: \"kubernetes.io/projected/d8724e53-459c-4708-aa69-9ca66d0f90ab-kube-api-access-6g9qf\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.455739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2844\" (UniqueName: \"kubernetes.io/projected/aef3e33d-1cbf-41e5-97a7-d821055fc115-kube-api-access-d2844\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.455780 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-system-cni-dir\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.455828 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-etc-kubernetes\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.455939 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-sysconfig\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.456680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456010 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-var-lib-kubelet\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456090 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-systemd\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456147 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3bcc9b3b-8cfd-4704-884c-36d40629bf51-lib-modules\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456206 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/423a3192-16a1-4e66-94de-9d88c359469c-dbus\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456267 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-run-netns\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456320 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-tuned\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456454 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-run-netns\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456482 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-var-lib-kubelet\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456525 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-run-ovn\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456604 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-node-log\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456772 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/444b4377-ba44-4941-9f99-1136ba12a661-host-slash\") pod \"iptables-alerter-8j7nt\" (UID: \"444b4377-ba44-4941-9f99-1136ba12a661\") " pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.454875 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/598c2c71-e257-4445-8f36-710b469c366f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456851 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9585l\" (UniqueName: \"kubernetes.io/projected/b89320cc-3dc2-406c-aa36-62aa5e8f377b-kube-api-access-9585l\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456886 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-var-lib-openvswitch\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456898 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/444b4377-ba44-4941-9f99-1136ba12a661-host-slash\") pod \"iptables-alerter-8j7nt\" (UID: \"444b4377-ba44-4941-9f99-1136ba12a661\") " pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.457467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.456919 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-cni-bin\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.460372 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.460061 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3bcc9b3b-8cfd-4704-884c-36d40629bf51-tmp\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.461360 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.461332 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd8720ca-7ec0-425b-9b19-2afaf1699d4d-agent-certs\") pod \"konnectivity-agent-x2dvn\" (UID: \"dd8720ca-7ec0-425b-9b19-2afaf1699d4d\") " pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:09:38.463827 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.463752 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3bcc9b3b-8cfd-4704-884c-36d40629bf51-etc-tuned\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.464513 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.464488 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8bb\" (UniqueName: \"kubernetes.io/projected/eaebfdaf-a012-490a-8e17-b8346cbf6748-kube-api-access-bx8bb\") pod \"node-ca-xqhsh\" (UID: \"eaebfdaf-a012-490a-8e17-b8346cbf6748\") " pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.464611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.464488 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4vg\" (UniqueName: \"kubernetes.io/projected/598c2c71-e257-4445-8f36-710b469c366f-kube-api-access-8d4vg\") pod \"multus-additional-cni-plugins-xlh9g\" (UID: \"598c2c71-e257-4445-8f36-710b469c366f\") " pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.464678 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.464611 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxdr\" (UniqueName: \"kubernetes.io/projected/444b4377-ba44-4941-9f99-1136ba12a661-kube-api-access-cvxdr\") pod \"iptables-alerter-8j7nt\" (UID: \"444b4377-ba44-4941-9f99-1136ba12a661\") " pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.465075 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.465057 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2844\" (UniqueName: \"kubernetes.io/projected/aef3e33d-1cbf-41e5-97a7-d821055fc115-kube-api-access-d2844\") pod \"aws-ebs-csi-driver-node-jp2kh\" (UID: \"aef3e33d-1cbf-41e5-97a7-d821055fc115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.465866 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.465844 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrpl\" (UniqueName: \"kubernetes.io/projected/3bcc9b3b-8cfd-4704-884c-36d40629bf51-kube-api-access-6qrpl\") pod \"tuned-czg45\" (UID: \"3bcc9b3b-8cfd-4704-884c-36d40629bf51\") " pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.528567 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.528511 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" event={"ID":"69103f5be0b7c8797e5968b28b95762b","Type":"ContainerStarted","Data":"4d6f37d6b8e3f0ad90a281ae781af4cf5015793b96d7e2221ee4b806360611d3"} Jul 06 19:09:38.529965 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.529937 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"6749d73f2a1c44df670398dd9fe19471","Type":"ContainerStarted","Data":"5b332baa6d236ab83ea3d88883b19f989256202f108c1587ada53d8bb11b75d7"} Jul 06 19:09:38.558138 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558104 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-kubelet\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558305 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558149 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-run-openvswitch\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558305 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558175 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g9qf\" (UniqueName: \"kubernetes.io/projected/d8724e53-459c-4708-aa69-9ca66d0f90ab-kube-api-access-6g9qf\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558305 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558200 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-system-cni-dir\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.558305 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558243 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-etc-kubernetes\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.558305 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558270 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-run-netns\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558305 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558296 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-run-netns\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558321 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-var-lib-kubelet\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558347 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-run-ovn\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558372 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-node-log\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558392 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9585l\" (UniqueName: \"kubernetes.io/projected/b89320cc-3dc2-406c-aa36-62aa5e8f377b-kube-api-access-9585l\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558415 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-var-lib-openvswitch\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558435 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-cni-bin\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558451 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-os-release\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558472 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-cni-netd\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558498 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8724e53-459c-4708-aa69-9ca66d0f90ab-ovnkube-config\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558527 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-var-lib-cni-bin\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558554 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-daemon-config\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558580 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.558611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558606 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-run-k8s-cni-cncf-io\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558639 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8724e53-459c-4708-aa69-9ca66d0f90ab-ovn-node-metrics-cert\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558665 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1c2a219-c8c6-4937-952e-630b3eb7de5a-hosts-file\") pod \"node-resolver-56v7k\" (UID: \"c1c2a219-c8c6-4937-952e-630b3eb7de5a\") " pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558688 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-socket-dir-parent\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558716 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8724e53-459c-4708-aa69-9ca66d0f90ab-ovnkube-script-lib\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558744 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6h8\" (UniqueName: \"kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8\") pod \"network-check-target-g5gwx\" (UID: \"9b975cfc-6a57-4352-8833-61170b28d14b\") " pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558755 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-kubelet\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558768 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-run-netns\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558790 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-var-lib-kubelet\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558822 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-var-lib-cni-bin\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558840 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-conf-dir\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558850 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-run-openvswitch\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558868 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-systemd-units\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558894 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-log-socket\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558919 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8724e53-459c-4708-aa69-9ca66d0f90ab-env-overrides\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558963 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-cni-dir\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.558987 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-var-lib-cni-multus\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559098 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559018 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559035 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-run-ovn\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559053 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b89320cc-3dc2-406c-aa36-62aa5e8f377b-cni-binary-copy\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559080 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-node-log\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559082 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-slash\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559114 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-slash\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559118 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-cnibin\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559135 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-system-cni-dir\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559149 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-run-multus-certs\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559166 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-cnibin\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559177 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pspnq\" (UniqueName: \"kubernetes.io/projected/c1c2a219-c8c6-4937-952e-630b3eb7de5a-kube-api-access-pspnq\") pod \"node-resolver-56v7k\" (UID: \"c1c2a219-c8c6-4937-952e-630b3eb7de5a\") " pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559181 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-etc-kubernetes\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559202 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-etc-openvswitch\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559212 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-cni-bin\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559229 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbcj\" (UniqueName: \"kubernetes.io/projected/ee50f9b8-fc16-4951-a486-88d460005a0c-kube-api-access-9mbcj\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559245 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-cni-dir\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559252 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-var-lib-openvswitch\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559263 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-run-systemd\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.559943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559294 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-conf-dir\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559295 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-run-netns\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559311 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559328 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-log-socket\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559338 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c2a219-c8c6-4937-952e-630b3eb7de5a-tmp-dir\") pod \"node-resolver-56v7k\" (UID: \"c1c2a219-c8c6-4937-952e-630b3eb7de5a\") " pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559360 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-systemd-units\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559371 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-hostroot\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559393 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-daemon-config\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559447 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-hostroot\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559455 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d8724e53-459c-4708-aa69-9ca66d0f90ab-ovnkube-config\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559462 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559487 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-etc-openvswitch\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559527 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-cni-netd\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559566 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559592 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-os-release\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559646 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c1c2a219-c8c6-4937-952e-630b3eb7de5a-hosts-file\") pod \"node-resolver-56v7k\" (UID: \"c1c2a219-c8c6-4937-952e-630b3eb7de5a\") " pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.559737 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559755 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d8724e53-459c-4708-aa69-9ca66d0f90ab-env-overrides\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.560860 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.559826 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs podName:ee50f9b8-fc16-4951-a486-88d460005a0c nodeName:}" failed. No retries permitted until 2026-07-06 19:09:39.05978679 +0000 UTC m=+4.543010344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs") pod "network-metrics-daemon-hlrkz" (UID: "ee50f9b8-fc16-4951-a486-88d460005a0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:38.561747 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559833 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d8724e53-459c-4708-aa69-9ca66d0f90ab-run-systemd\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.561747 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559899 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-run-k8s-cni-cncf-io\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.561747 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559957 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-multus-socket-dir-parent\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.561747 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.559996 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-var-lib-cni-multus\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.561747 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.560033 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d8724e53-459c-4708-aa69-9ca66d0f90ab-ovnkube-script-lib\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.561747 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.560087 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b89320cc-3dc2-406c-aa36-62aa5e8f377b-host-run-multus-certs\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.561747 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.560100 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b89320cc-3dc2-406c-aa36-62aa5e8f377b-cni-binary-copy\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.561747 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.560155 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c2a219-c8c6-4937-952e-630b3eb7de5a-tmp-dir\") pod \"node-resolver-56v7k\" (UID: \"c1c2a219-c8c6-4937-952e-630b3eb7de5a\") " pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.562552 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.562527 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8724e53-459c-4708-aa69-9ca66d0f90ab-ovn-node-metrics-cert\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.566525 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.566500 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:09:38.566525 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.566528 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:09:38.566687 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.566543 2554 projected.go:196] Error preparing data for projected volume kube-api-access-4j6h8 for pod openshift-network-diagnostics/network-check-target-g5gwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:38.566687 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.566617 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8 podName:9b975cfc-6a57-4352-8833-61170b28d14b nodeName:}" failed. No retries permitted until 2026-07-06 19:09:39.066596815 +0000 UTC m=+4.549820386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4j6h8" (UniqueName: "kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8") pod "network-check-target-g5gwx" (UID: "9b975cfc-6a57-4352-8833-61170b28d14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:38.569365 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.569326 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pspnq\" (UniqueName: \"kubernetes.io/projected/c1c2a219-c8c6-4937-952e-630b3eb7de5a-kube-api-access-pspnq\") pod \"node-resolver-56v7k\" (UID: \"c1c2a219-c8c6-4937-952e-630b3eb7de5a\") " pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.569489 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.569443 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbcj\" (UniqueName: \"kubernetes.io/projected/ee50f9b8-fc16-4951-a486-88d460005a0c-kube-api-access-9mbcj\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:38.569717 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.569684 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g9qf\" (UniqueName: \"kubernetes.io/projected/d8724e53-459c-4708-aa69-9ca66d0f90ab-kube-api-access-6g9qf\") pod \"ovnkube-node-bfz6c\" (UID: \"d8724e53-459c-4708-aa69-9ca66d0f90ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.570828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.570805 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9585l\" (UniqueName: \"kubernetes.io/projected/b89320cc-3dc2-406c-aa36-62aa5e8f377b-kube-api-access-9585l\") pod \"multus-kj27t\" (UID: \"b89320cc-3dc2-406c-aa36-62aa5e8f377b\") " pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.640774 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.640733 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xqhsh" Jul 06 19:09:38.651681 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.651651 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:09:38.661286 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.661263 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" Jul 06 19:09:38.667311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.667289 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-czg45" Jul 06 19:09:38.677086 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.677059 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8j7nt" Jul 06 19:09:38.685102 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.685075 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" Jul 06 19:09:38.692345 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.692312 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-56v7k" Jul 06 19:09:38.701266 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.701237 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kj27t" Jul 06 19:09:38.708082 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.708057 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:09:38.962889 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:38.962860 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:38.963038 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.962977 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:38.963038 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:38.963035 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret podName:423a3192-16a1-4e66-94de-9d88c359469c nodeName:}" failed. No retries permitted until 2026-07-06 19:09:39.963021158 +0000 UTC m=+5.446244711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret") pod "global-pull-secret-syncer-ncc6d" (UID: "423a3192-16a1-4e66-94de-9d88c359469c") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:39.063200 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:39.063169 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:39.063400 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:39.063341 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:39.063489 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:39.063408 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs podName:ee50f9b8-fc16-4951-a486-88d460005a0c nodeName:}" failed. No retries permitted until 2026-07-06 19:09:40.063388275 +0000 UTC m=+5.546611828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs") pod "network-metrics-daemon-hlrkz" (UID: "ee50f9b8-fc16-4951-a486-88d460005a0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:39.164121 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:39.164080 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6h8\" (UniqueName: \"kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8\") pod \"network-check-target-g5gwx\" (UID: \"9b975cfc-6a57-4352-8833-61170b28d14b\") " pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:39.164313 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:39.164277 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:09:39.164313 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:39.164301 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:09:39.164313 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:39.164314 2554 projected.go:196] Error preparing data for projected volume kube-api-access-4j6h8 for pod openshift-network-diagnostics/network-check-target-g5gwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:39.164474 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:39.164402 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8 podName:9b975cfc-6a57-4352-8833-61170b28d14b nodeName:}" failed. No retries permitted until 2026-07-06 19:09:40.164364168 +0000 UTC m=+5.647587736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j6h8" (UniqueName: "kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8") pod "network-check-target-g5gwx" (UID: "9b975cfc-6a57-4352-8833-61170b28d14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:39.383285 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:39.383184 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:04:38 +0000 UTC" deadline="2028-02-29 15:56:00.538972402 +0000 UTC" Jul 06 19:09:39.383285 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:39.383231 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14468h46m21.15574623s" Jul 06 19:09:39.524364 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:39.524328 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:39.524539 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:39.524366 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:39.524539 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:39.524481 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:39.524674 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:39.524623 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:39.757658 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:39.757620 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8720ca_7ec0_425b_9b19_2afaf1699d4d.slice/crio-42aa685467b4f60a63e23427ff3a52ed96818f9c3f8f1b2f12193f5286055243 WatchSource:0}: Error finding container 42aa685467b4f60a63e23427ff3a52ed96818f9c3f8f1b2f12193f5286055243: Status 404 returned error can't find the container with id 42aa685467b4f60a63e23427ff3a52ed96818f9c3f8f1b2f12193f5286055243 Jul 06 19:09:39.759016 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:39.758978 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8724e53_459c_4708_aa69_9ca66d0f90ab.slice/crio-8dd47cc0625224db7dcf7ddb5e6e75db26935c7192e0c9020844b415d8b91f1a WatchSource:0}: Error finding container 8dd47cc0625224db7dcf7ddb5e6e75db26935c7192e0c9020844b415d8b91f1a: Status 404 returned error can't find the container with id 8dd47cc0625224db7dcf7ddb5e6e75db26935c7192e0c9020844b415d8b91f1a Jul 06 19:09:39.764209 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:39.764183 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598c2c71_e257_4445_8f36_710b469c366f.slice/crio-52a68d514d264507ec49d0bead1a42ea904ebb3fab51955a3d6bb5299b21fac6 WatchSource:0}: Error finding container 52a68d514d264507ec49d0bead1a42ea904ebb3fab51955a3d6bb5299b21fac6: Status 404 returned error can't find the container with id 52a68d514d264507ec49d0bead1a42ea904ebb3fab51955a3d6bb5299b21fac6 Jul 06 19:09:39.765769 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:39.765728 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaebfdaf_a012_490a_8e17_b8346cbf6748.slice/crio-7a96dbd980003eaa31a0dcd1441f63fe541563307e5d159a4dea9632628d2081 WatchSource:0}: Error finding container 7a96dbd980003eaa31a0dcd1441f63fe541563307e5d159a4dea9632628d2081: Status 404 returned error can't find the container with id 7a96dbd980003eaa31a0dcd1441f63fe541563307e5d159a4dea9632628d2081 Jul 06 19:09:39.766680 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:39.766638 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef3e33d_1cbf_41e5_97a7_d821055fc115.slice/crio-cfb1be15324585e98978a54a0f5331ebb2c7164f9205abea9c011ed1cd7790c2 WatchSource:0}: Error finding container cfb1be15324585e98978a54a0f5331ebb2c7164f9205abea9c011ed1cd7790c2: Status 404 returned error can't find the container with id cfb1be15324585e98978a54a0f5331ebb2c7164f9205abea9c011ed1cd7790c2 Jul 06 19:09:39.768838 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:39.768657 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89320cc_3dc2_406c_aa36_62aa5e8f377b.slice/crio-f9ebdf5bf0189f69e661e564d0159ae4b028af947494d4fef0119d1e4b7bba14 WatchSource:0}: Error finding container f9ebdf5bf0189f69e661e564d0159ae4b028af947494d4fef0119d1e4b7bba14: Status 404 returned error can't find the container with id f9ebdf5bf0189f69e661e564d0159ae4b028af947494d4fef0119d1e4b7bba14 Jul 06 19:09:39.773566 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:39.772008 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444b4377_ba44_4941_9f99_1136ba12a661.slice/crio-9c85f192a6d296dd8faf3f3b2cd3bbe3bdcfbff8981e7790b93630de53c55f32 WatchSource:0}: Error finding container 9c85f192a6d296dd8faf3f3b2cd3bbe3bdcfbff8981e7790b93630de53c55f32: Status 404 returned error can't find the container with id 9c85f192a6d296dd8faf3f3b2cd3bbe3bdcfbff8981e7790b93630de53c55f32 Jul 06 19:09:39.773566 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:39.772222 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcc9b3b_8cfd_4704_884c_36d40629bf51.slice/crio-92af2bbc4d06b9e8762fb16d6a76474d860864f721c5a96ed7ba034b81c4c8aa WatchSource:0}: Error finding container 92af2bbc4d06b9e8762fb16d6a76474d860864f721c5a96ed7ba034b81c4c8aa: Status 404 returned error can't find the container with id 92af2bbc4d06b9e8762fb16d6a76474d860864f721c5a96ed7ba034b81c4c8aa Jul 06 19:09:39.776509 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:09:39.776478 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c2a219_c8c6_4937_952e_630b3eb7de5a.slice/crio-cb373df7d61b070b5dc62dacd259f10c2059fa81ec04ccc77ac61cc5e3bf778d WatchSource:0}: Error finding container cb373df7d61b070b5dc62dacd259f10c2059fa81ec04ccc77ac61cc5e3bf778d: Status 404 returned error can't find the container with id cb373df7d61b070b5dc62dacd259f10c2059fa81ec04ccc77ac61cc5e3bf778d Jul 06 19:09:39.970974 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:39.970739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:39.971132 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:39.970867 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:39.971132 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:39.971049 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret podName:423a3192-16a1-4e66-94de-9d88c359469c nodeName:}" failed. No retries permitted until 2026-07-06 19:09:41.971030968 +0000 UTC m=+7.454254522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret") pod "global-pull-secret-syncer-ncc6d" (UID: "423a3192-16a1-4e66-94de-9d88c359469c") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:40.071976 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.071952 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:40.072086 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:40.072073 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:40.072136 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:40.072130 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs podName:ee50f9b8-fc16-4951-a486-88d460005a0c nodeName:}" failed. No retries permitted until 2026-07-06 19:09:42.072115173 +0000 UTC m=+7.555338725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs") pod "network-metrics-daemon-hlrkz" (UID: "ee50f9b8-fc16-4951-a486-88d460005a0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:40.172330 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.172295 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6h8\" (UniqueName: \"kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8\") pod \"network-check-target-g5gwx\" (UID: \"9b975cfc-6a57-4352-8833-61170b28d14b\") " pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:40.172546 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:40.172461 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:09:40.172546 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:40.172486 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:09:40.172546 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:40.172497 2554 projected.go:196] Error preparing data for projected volume kube-api-access-4j6h8 for pod openshift-network-diagnostics/network-check-target-g5gwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:40.172830 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:40.172551 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8 podName:9b975cfc-6a57-4352-8833-61170b28d14b nodeName:}" failed. No retries permitted until 2026-07-06 19:09:42.172528827 +0000 UTC m=+7.655752382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j6h8" (UniqueName: "kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8") pod "network-check-target-g5gwx" (UID: "9b975cfc-6a57-4352-8833-61170b28d14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:40.384181 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.384065 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:04:38 +0000 UTC" deadline="2028-02-04 00:20:06.497575685 +0000 UTC" Jul 06 19:09:40.384181 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.384131 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13853h10m26.113449049s" Jul 06 19:09:40.524057 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.524021 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:40.524243 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:40.524166 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:40.553349 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.553311 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" event={"ID":"69103f5be0b7c8797e5968b28b95762b","Type":"ContainerStarted","Data":"02e15b8240c801c61f347579a34be932a42fc33a3481d3f24d5a5d29f00555e0"} Jul 06 19:09:40.579799 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.579763 2554 generic.go:356] "Generic (PLEG): container finished" podID="6749d73f2a1c44df670398dd9fe19471" containerID="d7a56a1d94baadab173faf40d29eaeee55cda8cbbd6bf1858e0a3c3d2208cc4b" exitCode=0 Jul 06 19:09:40.580155 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.580130 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"6749d73f2a1c44df670398dd9fe19471","Type":"ContainerDied","Data":"d7a56a1d94baadab173faf40d29eaeee55cda8cbbd6bf1858e0a3c3d2208cc4b"} Jul 06 19:09:40.590538 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.590509 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-56v7k" event={"ID":"c1c2a219-c8c6-4937-952e-630b3eb7de5a","Type":"ContainerStarted","Data":"cb373df7d61b070b5dc62dacd259f10c2059fa81ec04ccc77ac61cc5e3bf778d"} Jul 06 19:09:40.593658 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.593602 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kj27t" event={"ID":"b89320cc-3dc2-406c-aa36-62aa5e8f377b","Type":"ContainerStarted","Data":"f9ebdf5bf0189f69e661e564d0159ae4b028af947494d4fef0119d1e4b7bba14"} Jul 06 19:09:40.596078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.595987 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" event={"ID":"aef3e33d-1cbf-41e5-97a7-d821055fc115","Type":"ContainerStarted","Data":"cfb1be15324585e98978a54a0f5331ebb2c7164f9205abea9c011ed1cd7790c2"} Jul 06 19:09:40.598719 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.598672 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" podStartSLOduration=3.598654498 podStartE2EDuration="3.598654498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:40.568852975 +0000 UTC m=+6.052076572" watchObservedRunningTime="2026-07-06 19:09:40.598654498 +0000 UTC m=+6.081878072" Jul 06 19:09:40.604050 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.604022 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" event={"ID":"598c2c71-e257-4445-8f36-710b469c366f","Type":"ContainerStarted","Data":"52a68d514d264507ec49d0bead1a42ea904ebb3fab51955a3d6bb5299b21fac6"} Jul 06 19:09:40.613957 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.613931 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" event={"ID":"d8724e53-459c-4708-aa69-9ca66d0f90ab","Type":"ContainerStarted","Data":"8dd47cc0625224db7dcf7ddb5e6e75db26935c7192e0c9020844b415d8b91f1a"} Jul 06 19:09:40.625479 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.625452 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x2dvn" event={"ID":"dd8720ca-7ec0-425b-9b19-2afaf1699d4d","Type":"ContainerStarted","Data":"42aa685467b4f60a63e23427ff3a52ed96818f9c3f8f1b2f12193f5286055243"} Jul 06 19:09:40.655586 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.655507 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-czg45" event={"ID":"3bcc9b3b-8cfd-4704-884c-36d40629bf51","Type":"ContainerStarted","Data":"92af2bbc4d06b9e8762fb16d6a76474d860864f721c5a96ed7ba034b81c4c8aa"} Jul 06 19:09:40.671530 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.671497 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8j7nt" event={"ID":"444b4377-ba44-4941-9f99-1136ba12a661","Type":"ContainerStarted","Data":"9c85f192a6d296dd8faf3f3b2cd3bbe3bdcfbff8981e7790b93630de53c55f32"} Jul 06 19:09:40.697343 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:40.697302 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xqhsh" event={"ID":"eaebfdaf-a012-490a-8e17-b8346cbf6748","Type":"ContainerStarted","Data":"7a96dbd980003eaa31a0dcd1441f63fe541563307e5d159a4dea9632628d2081"} Jul 06 19:09:41.524131 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:41.524096 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:41.524593 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:41.524241 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:41.524667 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:41.524616 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:41.524721 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:41.524706 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:41.734972 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:41.734938 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"6749d73f2a1c44df670398dd9fe19471","Type":"ContainerStarted","Data":"d7eabd317ac32f167ae6575bd5d2bd8c0c0d709ffb43bbcfc89a546f60566eb9"} Jul 06 19:09:41.987348 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:41.987260 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:41.987640 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:41.987396 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:41.987640 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:41.987460 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret podName:423a3192-16a1-4e66-94de-9d88c359469c nodeName:}" failed. No retries permitted until 2026-07-06 19:09:45.987441847 +0000 UTC m=+11.470665404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret") pod "global-pull-secret-syncer-ncc6d" (UID: "423a3192-16a1-4e66-94de-9d88c359469c") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:42.088039 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:42.088004 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:42.088277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:42.088050 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:42.088277 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:42.088094 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs podName:ee50f9b8-fc16-4951-a486-88d460005a0c nodeName:}" failed. No retries permitted until 2026-07-06 19:09:46.088073892 +0000 UTC m=+11.571297463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs") pod "network-metrics-daemon-hlrkz" (UID: "ee50f9b8-fc16-4951-a486-88d460005a0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:42.189543 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:42.189504 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6h8\" (UniqueName: \"kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8\") pod \"network-check-target-g5gwx\" (UID: \"9b975cfc-6a57-4352-8833-61170b28d14b\") " pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:42.189768 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:42.189676 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:09:42.189768 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:42.189693 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:09:42.189768 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:42.189709 2554 projected.go:196] Error preparing data for projected volume kube-api-access-4j6h8 for pod openshift-network-diagnostics/network-check-target-g5gwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:42.189932 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:42.189783 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8 podName:9b975cfc-6a57-4352-8833-61170b28d14b nodeName:}" failed. No retries permitted until 2026-07-06 19:09:46.189764704 +0000 UTC m=+11.672988269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j6h8" (UniqueName: "kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8") pod "network-check-target-g5gwx" (UID: "9b975cfc-6a57-4352-8833-61170b28d14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:42.523782 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:42.523751 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:42.523955 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:42.523904 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:43.524472 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:43.523958 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:43.524472 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:43.523959 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:43.524472 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:43.524100 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:43.524472 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:43.524193 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:44.524500 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:44.524465 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:44.524997 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:44.524603 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:45.524397 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:45.524082 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:45.524397 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:45.524117 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:45.524397 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:45.524244 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:45.525083 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:45.524642 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:46.020176 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:46.020085 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:46.020335 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:46.020265 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:46.020335 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:46.020328 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret podName:423a3192-16a1-4e66-94de-9d88c359469c nodeName:}" failed. No retries permitted until 2026-07-06 19:09:54.020309491 +0000 UTC m=+19.503533046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret") pod "global-pull-secret-syncer-ncc6d" (UID: "423a3192-16a1-4e66-94de-9d88c359469c") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:46.120677 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:46.120642 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:46.120869 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:46.120820 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:46.120932 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:46.120892 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs podName:ee50f9b8-fc16-4951-a486-88d460005a0c nodeName:}" failed. No retries permitted until 2026-07-06 19:09:54.120871366 +0000 UTC m=+19.604094918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs") pod "network-metrics-daemon-hlrkz" (UID: "ee50f9b8-fc16-4951-a486-88d460005a0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:46.222153 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:46.221796 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6h8\" (UniqueName: \"kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8\") pod \"network-check-target-g5gwx\" (UID: \"9b975cfc-6a57-4352-8833-61170b28d14b\") " pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:46.222153 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:46.222046 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:09:46.222153 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:46.222071 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:09:46.222153 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:46.222084 2554 projected.go:196] Error preparing data for projected volume kube-api-access-4j6h8 for pod openshift-network-diagnostics/network-check-target-g5gwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:46.222153 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:46.222142 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8 podName:9b975cfc-6a57-4352-8833-61170b28d14b nodeName:}" failed. No retries permitted until 2026-07-06 19:09:54.222123846 +0000 UTC m=+19.705347400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j6h8" (UniqueName: "kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8") pod "network-check-target-g5gwx" (UID: "9b975cfc-6a57-4352-8833-61170b28d14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:46.524994 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:46.524960 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:46.525411 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:46.525081 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:47.523968 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:47.523931 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:47.524138 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:47.523949 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:47.524138 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:47.524084 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:47.524254 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:47.524184 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:48.523965 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:48.523936 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:48.524412 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:48.524053 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:49.523994 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:49.523962 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:49.524416 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:49.524011 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:49.524416 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:49.524112 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:49.524416 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:49.524232 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:50.524315 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:50.524278 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:50.524852 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:50.524417 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:51.524533 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:51.524492 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:51.524533 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:51.524522 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:51.525069 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:51.524635 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:51.525069 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:51.524775 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:52.524386 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:52.524350 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:52.524591 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:52.524493 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:53.524088 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:53.523997 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:53.524088 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:53.524033 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:53.524324 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:53.524153 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:53.524324 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:53.524284 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:54.085420 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:54.085383 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:54.085842 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:54.085548 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:54.085842 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:54.085623 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret podName:423a3192-16a1-4e66-94de-9d88c359469c nodeName:}" failed. No retries permitted until 2026-07-06 19:10:10.085603657 +0000 UTC m=+35.568827245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret") pod "global-pull-secret-syncer-ncc6d" (UID: "423a3192-16a1-4e66-94de-9d88c359469c") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:09:54.186448 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:54.186407 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:54.186589 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:54.186559 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:54.186630 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:54.186625 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs podName:ee50f9b8-fc16-4951-a486-88d460005a0c nodeName:}" failed. No retries permitted until 2026-07-06 19:10:10.186609061 +0000 UTC m=+35.669832616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs") pod "network-metrics-daemon-hlrkz" (UID: "ee50f9b8-fc16-4951-a486-88d460005a0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:09:54.287104 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:54.287067 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6h8\" (UniqueName: \"kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8\") pod \"network-check-target-g5gwx\" (UID: \"9b975cfc-6a57-4352-8833-61170b28d14b\") " pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:54.287266 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:54.287225 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:09:54.287266 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:54.287244 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:09:54.287266 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:54.287256 2554 projected.go:196] Error preparing data for projected volume kube-api-access-4j6h8 for pod openshift-network-diagnostics/network-check-target-g5gwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:54.287422 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:54.287322 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8 podName:9b975cfc-6a57-4352-8833-61170b28d14b nodeName:}" failed. No retries permitted until 2026-07-06 19:10:10.28730218 +0000 UTC m=+35.770525745 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j6h8" (UniqueName: "kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8") pod "network-check-target-g5gwx" (UID: "9b975cfc-6a57-4352-8833-61170b28d14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:09:54.524259 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:54.524176 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:54.524416 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:54.524326 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:55.523996 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:55.523964 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:55.524442 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:55.523969 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:55.524442 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:55.524110 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:55.524442 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:55.524225 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:56.525174 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:56.525135 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:56.525657 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:56.525255 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:57.524662 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:57.524625 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:57.524845 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:57.524630 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:57.524845 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:57.524755 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:57.524920 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:57.524840 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:58.524083 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.523842 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:09:58.525018 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:58.524169 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:09:58.775018 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.774983 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-czg45" event={"ID":"3bcc9b3b-8cfd-4704-884c-36d40629bf51","Type":"ContainerStarted","Data":"bd4a1a7fc73514570db8ad8ebba42c8ba67e44ac098bbc93b8d397054f6f41c8"} Jul 06 19:09:58.776346 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.776321 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xqhsh" event={"ID":"eaebfdaf-a012-490a-8e17-b8346cbf6748","Type":"ContainerStarted","Data":"b9cdea66f20765d649b74ceb1184c57f4acbcd6b2db93eaa81d6a6d4d46cbe8c"} Jul 06 19:09:58.777484 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.777464 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-56v7k" event={"ID":"c1c2a219-c8c6-4937-952e-630b3eb7de5a","Type":"ContainerStarted","Data":"2fc01eae2b925b9bf75d19f85afd840dd897c8c843fdae4be49c63979d1fddb2"} Jul 06 19:09:58.778725 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.778701 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kj27t" event={"ID":"b89320cc-3dc2-406c-aa36-62aa5e8f377b","Type":"ContainerStarted","Data":"cf611d4d7110e09660767da0b990e37850c130d03490074192720d7cc17cb4e2"} Jul 06 19:09:58.779982 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.779963 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" event={"ID":"aef3e33d-1cbf-41e5-97a7-d821055fc115","Type":"ContainerStarted","Data":"7e98dbb9a7a1f0f5161a5f93636664b811101bb26beb21c6e47f53aef223c5ef"} Jul 06 19:09:58.781289 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.781270 2554 generic.go:356] "Generic (PLEG): container finished" podID="598c2c71-e257-4445-8f36-710b469c366f" containerID="2d176db6260241ea91ae557ab1073d8783f1c9dbcca07dba52f400739e2a9a11" exitCode=0 Jul 06 19:09:58.781354 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.781335 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" event={"ID":"598c2c71-e257-4445-8f36-710b469c366f","Type":"ContainerDied","Data":"2d176db6260241ea91ae557ab1073d8783f1c9dbcca07dba52f400739e2a9a11"} Jul 06 19:09:58.783808 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.783792 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:09:58.784107 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.784089 2554 generic.go:356] "Generic (PLEG): container finished" podID="d8724e53-459c-4708-aa69-9ca66d0f90ab" containerID="8534cb2d60971ba62ef142b8ce75c76b74e73f201bb464ff262ecceebc5d9d91" exitCode=1 Jul 06 19:09:58.784177 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.784159 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" event={"ID":"d8724e53-459c-4708-aa69-9ca66d0f90ab","Type":"ContainerStarted","Data":"24f969dc27bd8e55a108785bc1de5e81d28226cedb971d8ef15fdb67bb1dbc0b"} Jul 06 19:09:58.784228 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.784215 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" event={"ID":"d8724e53-459c-4708-aa69-9ca66d0f90ab","Type":"ContainerStarted","Data":"6f11bc7ab3e34742ba53e1bc5e4e278d4cbee9b4e0a15707522491158d503e10"} Jul 06 19:09:58.784275 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.784260 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" event={"ID":"d8724e53-459c-4708-aa69-9ca66d0f90ab","Type":"ContainerStarted","Data":"34a45361175b45f42425ef64fa1da3c59717e3a6d5ff2c8a4c9b842c76829ebe"} Jul 06 19:09:58.784317 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.784288 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" event={"ID":"d8724e53-459c-4708-aa69-9ca66d0f90ab","Type":"ContainerStarted","Data":"68309a17c5bf3d1b10b8b048a25d959f21122aaabccbcbc95cd0a949b0c12a50"} Jul 06 19:09:58.784362 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.784316 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" event={"ID":"d8724e53-459c-4708-aa69-9ca66d0f90ab","Type":"ContainerDied","Data":"8534cb2d60971ba62ef142b8ce75c76b74e73f201bb464ff262ecceebc5d9d91"} Jul 06 19:09:58.784362 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.784350 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" event={"ID":"d8724e53-459c-4708-aa69-9ca66d0f90ab","Type":"ContainerStarted","Data":"873ce41c15c52066aabf2a359b06198e6622b49d493d4004d9aac7db7f9efa30"} Jul 06 19:09:58.785329 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.785314 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x2dvn" event={"ID":"dd8720ca-7ec0-425b-9b19-2afaf1699d4d","Type":"ContainerStarted","Data":"f9399580ead57bf408c811b70d81d7030d1e0f32064bdcfb9d1dd78ad0bbdea6"} Jul 06 19:09:58.790913 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.790880 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-czg45" podStartSLOduration=4.942304375 podStartE2EDuration="22.790868012s" totalImagesPullingTime="17.848563637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:58.790806384 +0000 UTC m=+24.274029956" watchObservedRunningTime="2026-07-06 19:09:58.790868012 +0000 UTC m=+24.274091586" Jul 06 19:09:58.790986 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.790963 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" podStartSLOduration=21.790958178 podStartE2EDuration="21.790958178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:41.772444759 +0000 UTC m=+7.255668336" watchObservedRunningTime="2026-07-06 19:09:58.790958178 +0000 UTC m=+24.274181750" Jul 06 19:09:58.805744 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.805686 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-56v7k" podStartSLOduration=3.978191891 podStartE2EDuration="21.805675838s" totalImagesPullingTime="17.827483947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:58.805410597 +0000 UTC m=+24.288634169" watchObservedRunningTime="2026-07-06 19:09:58.805675838 +0000 UTC m=+24.288899411" Jul 06 19:09:58.822353 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.822314 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kj27t" podStartSLOduration=3.926158132 podStartE2EDuration="21.822306492s" totalImagesPullingTime="17.89614836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:58.822265196 +0000 UTC m=+24.305488772" watchObservedRunningTime="2026-07-06 19:09:58.822306492 +0000 UTC m=+24.305530065" Jul 06 19:09:58.835909 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.835869 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xqhsh" podStartSLOduration=4.962492117 podStartE2EDuration="22.835862793s" totalImagesPullingTime="17.873370676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:58.835686154 +0000 UTC m=+24.318909726" watchObservedRunningTime="2026-07-06 19:09:58.835862793 +0000 UTC m=+24.319086365" Jul 06 19:09:58.849676 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.849639 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x2dvn" podStartSLOduration=9.43719154 podStartE2EDuration="22.84963228s" totalImagesPullingTime="13.41244074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:58.849418967 +0000 UTC m=+24.332642539" watchObservedRunningTime="2026-07-06 19:09:58.84963228 +0000 UTC m=+24.332855852" Jul 06 19:09:58.998107 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:58.998082 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:09:59.413343 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:59.413107 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:09:58.998101484Z","UUID":"5d789d49-8022-4ed1-9120-a199b13d930c","Handler":null,"Name":"","Endpoint":""} Jul 06 19:09:59.414858 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:59.414836 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:09:59.414992 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:59.414879 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:09:59.524688 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:59.524649 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:09:59.525154 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:59.524777 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:09:59.525154 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:59.524832 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:09:59.525154 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:09:59.524943 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:09:59.790395 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:59.790316 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" event={"ID":"aef3e33d-1cbf-41e5-97a7-d821055fc115","Type":"ContainerStarted","Data":"20319a817a4928d45547e79b1adc899089453b16343259a83953fbffc98b45fc"} Jul 06 19:09:59.791951 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:59.791891 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8j7nt" event={"ID":"444b4377-ba44-4941-9f99-1136ba12a661","Type":"ContainerStarted","Data":"22633f5bea63e89ba32734d44a918d83a3a862bd656d51911267d6e8a4a3b9ab"} Jul 06 19:09:59.808229 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:09:59.808179 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8j7nt" podStartSLOduration=5.959418941 podStartE2EDuration="23.808159485s" totalImagesPullingTime="17.848740544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:59.807740494 +0000 UTC m=+25.290964063" watchObservedRunningTime="2026-07-06 19:09:59.808159485 +0000 UTC m=+25.291383089" Jul 06 19:10:00.523853 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:00.523829 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:00.523981 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:00.523959 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:10:00.795444 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:00.795362 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" event={"ID":"aef3e33d-1cbf-41e5-97a7-d821055fc115","Type":"ContainerStarted","Data":"02209e3721e3dac015aadec512e326bbf272a081bf1510503a5dd545550bbdcf"} Jul 06 19:10:00.798280 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:00.798262 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:10:00.798541 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:00.798521 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" event={"ID":"d8724e53-459c-4708-aa69-9ca66d0f90ab","Type":"ContainerStarted","Data":"8cc3a1ae0db6a943c6618831d3bc7642b45e4aaa7d226bd2826cee2915f4b5ce"} Jul 06 19:10:00.813904 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:00.813859 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jp2kh" podStartSLOduration=4.8248769750000005 podStartE2EDuration="24.813844819s" totalImagesPullingTime="19.988967844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:00.81349435 +0000 UTC m=+26.296717923" watchObservedRunningTime="2026-07-06 19:10:00.813844819 +0000 UTC m=+26.297068391" Jul 06 19:10:01.524023 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:01.523991 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:01.524023 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:01.524009 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:01.524264 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:01.524123 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:10:01.524264 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:01.524237 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:10:02.524510 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:02.524482 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:02.524993 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:02.524607 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:10:02.807324 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:02.807076 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:10:02.807695 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:02.807663 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" event={"ID":"d8724e53-459c-4708-aa69-9ca66d0f90ab","Type":"ContainerStarted","Data":"5b87faaa0431de31d4636858214b028c9a5cafde68042d1b347308dcd90e24fe"} Jul 06 19:10:02.808039 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:02.808001 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:10:02.808179 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:02.808160 2554 scope.go:117] "RemoveContainer" containerID="8534cb2d60971ba62ef142b8ce75c76b74e73f201bb464ff262ecceebc5d9d91" Jul 06 19:10:02.824717 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:02.824697 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:10:03.523638 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.523604 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:03.523638 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.523633 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:03.523877 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:03.523706 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:10:03.523877 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:03.523851 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:10:03.543377 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.543342 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:10:03.544265 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.544253 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:10:03.820364 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.820337 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:10:03.820935 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.820914 2554 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 19:10:03.821103 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.821075 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" event={"ID":"d8724e53-459c-4708-aa69-9ca66d0f90ab","Type":"ContainerStarted","Data":"fadad2e721c46627b0c6c0c889d3c3dcebba6004cd20b82cba05fcc854fe516c"} Jul 06 19:10:03.821296 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.821190 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:10:03.821296 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.821235 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:10:03.821950 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.821921 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x2dvn" Jul 06 19:10:03.840903 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.840870 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:10:03.849051 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:03.848998 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" podStartSLOduration=8.907760401 podStartE2EDuration="26.848981947s" totalImagesPullingTime="17.941221546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:03.848426568 +0000 UTC m=+29.331650145" watchObservedRunningTime="2026-07-06 19:10:03.848981947 +0000 UTC m=+29.332205522" Jul 06 19:10:04.273629 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:04.273544 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:10:04.524166 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:04.524054 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:04.524349 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:04.524188 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:10:04.610726 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:04.610693 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ncc6d"] Jul 06 19:10:04.611174 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:04.611080 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5gwx"] Jul 06 19:10:04.611236 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:04.611177 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:04.611305 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:04.611283 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:10:04.611770 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:04.611747 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hlrkz"] Jul 06 19:10:04.611883 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:04.611851 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:04.611972 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:04.611953 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:10:04.823208 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:04.823177 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:04.823822 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:04.823797 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:10:06.524579 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:06.524491 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:06.524969 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:06.524576 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:10:06.524969 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:06.524620 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:06.524969 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:06.524620 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:06.524969 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:06.524767 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:10:06.524969 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:06.524783 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:10:06.829807 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:06.829770 2554 generic.go:356] "Generic (PLEG): container finished" podID="598c2c71-e257-4445-8f36-710b469c366f" containerID="dea99aa0ce5cd22bf1048c4996b0ded53c95c9f07a2d84497ab628dd4202e752" exitCode=0 Jul 06 19:10:06.829969 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:06.829831 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" event={"ID":"598c2c71-e257-4445-8f36-710b469c366f","Type":"ContainerDied","Data":"dea99aa0ce5cd22bf1048c4996b0ded53c95c9f07a2d84497ab628dd4202e752"} Jul 06 19:10:07.834030 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:07.833759 2554 generic.go:356] "Generic (PLEG): container finished" podID="598c2c71-e257-4445-8f36-710b469c366f" containerID="a23291f392960f3f2118466031509c1cf10a0dbf613e8abb7c5c251a22729e30" exitCode=0 Jul 06 19:10:07.834392 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:07.833835 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" event={"ID":"598c2c71-e257-4445-8f36-710b469c366f","Type":"ContainerDied","Data":"a23291f392960f3f2118466031509c1cf10a0dbf613e8abb7c5c251a22729e30"} Jul 06 19:10:08.523979 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:08.523945 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:08.524169 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:08.523945 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:08.524169 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:08.524082 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:08.524169 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:08.524077 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:10:08.524320 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:08.524173 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:10:08.524320 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:08.524264 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:10:08.838714 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:08.838682 2554 generic.go:356] "Generic (PLEG): container finished" podID="598c2c71-e257-4445-8f36-710b469c366f" containerID="f0fbad2e4ecc025967c890e3cbfae75c93ca80d57a02eb81a6eb3d706d8c6719" exitCode=0 Jul 06 19:10:08.839091 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:08.838764 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" event={"ID":"598c2c71-e257-4445-8f36-710b469c366f","Type":"ContainerDied","Data":"f0fbad2e4ecc025967c890e3cbfae75c93ca80d57a02eb81a6eb3d706d8c6719"} Jul 06 19:10:10.106604 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.106566 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:10.107247 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.106724 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:10:10.107247 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.106809 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret podName:423a3192-16a1-4e66-94de-9d88c359469c nodeName:}" failed. No retries permitted until 2026-07-06 19:10:42.10679178 +0000 UTC m=+67.590015350 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret") pod "global-pull-secret-syncer-ncc6d" (UID: "423a3192-16a1-4e66-94de-9d88c359469c") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:10:10.207267 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.207226 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:10.207533 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.207395 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:10:10.207533 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.207461 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs podName:ee50f9b8-fc16-4951-a486-88d460005a0c nodeName:}" failed. No retries permitted until 2026-07-06 19:10:42.207444494 +0000 UTC m=+67.690668046 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs") pod "network-metrics-daemon-hlrkz" (UID: "ee50f9b8-fc16-4951-a486-88d460005a0c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:10:10.308487 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.308455 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6h8\" (UniqueName: \"kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8\") pod \"network-check-target-g5gwx\" (UID: \"9b975cfc-6a57-4352-8833-61170b28d14b\") " pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:10.308671 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.308650 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:10:10.308760 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.308682 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:10:10.308760 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.308696 2554 projected.go:196] Error preparing data for projected volume kube-api-access-4j6h8 for pod openshift-network-diagnostics/network-check-target-g5gwx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:10:10.308861 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.308780 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8 podName:9b975cfc-6a57-4352-8833-61170b28d14b nodeName:}" failed. No retries permitted until 2026-07-06 19:10:42.30875787 +0000 UTC m=+67.791981438 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4j6h8" (UniqueName: "kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8") pod "network-check-target-g5gwx" (UID: "9b975cfc-6a57-4352-8833-61170b28d14b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:10:10.524746 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.524656 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:10.525007 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.524656 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:10.525007 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.524802 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncc6d" podUID="423a3192-16a1-4e66-94de-9d88c359469c" Jul 06 19:10:10.525007 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.524670 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:10.525007 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.524862 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5gwx" podUID="9b975cfc-6a57-4352-8833-61170b28d14b" Jul 06 19:10:10.525007 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:10.524950 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hlrkz" podUID="ee50f9b8-fc16-4951-a486-88d460005a0c" Jul 06 19:10:10.747356 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.747325 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeReady" Jul 06 19:10:10.747528 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.747466 2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:10:10.782020 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.781949 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84995c7cbf-v2kvf"] Jul 06 19:10:10.799002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.798976 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9x7qx"] Jul 06 19:10:10.799144 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.799113 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:10.801873 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.801833 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 19:10:10.802115 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.802093 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 19:10:10.802255 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.802236 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5jctw\"" Jul 06 19:10:10.802420 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.802403 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 19:10:10.808880 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.808860 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 19:10:10.816400 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.816377 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84995c7cbf-v2kvf"] Jul 06 19:10:10.816516 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.816407 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fpfnt"] Jul 06 19:10:10.816516 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.816496 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:10.819026 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.819005 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:10:10.819137 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.819068 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2k95g\"" Jul 06 19:10:10.819137 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.819073 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:10:10.833288 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.833265 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9x7qx"] Jul 06 19:10:10.833388 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.833315 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fpfnt" Jul 06 19:10:10.833388 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.833322 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fpfnt"] Jul 06 19:10:10.835930 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.835913 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:10:10.836039 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.835937 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:10:10.836039 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.835972 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:10:10.836039 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.836016 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5m8hp\"" Jul 06 19:10:10.913019 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.912984 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c7sb\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-kube-api-access-5c7sb\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:10.913019 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913024 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae485556-28dc-4ad6-98e2-7b41753bb66e-cert\") pod \"ingress-canary-fpfnt\" (UID: \"ae485556-28dc-4ad6-98e2-7b41753bb66e\") " pod="openshift-ingress-canary/ingress-canary-fpfnt" Jul 06 19:10:10.913251 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913044 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3da481-af3b-4827-95e5-bdd08e4ff256-tmp-dir\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:10.913251 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913063 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:10.913251 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913085 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db3da481-af3b-4827-95e5-bdd08e4ff256-config-volume\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:10.913251 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913137 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-installation-pull-secrets\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:10.913251 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913177 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-certificates\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:10.913251 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913196 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sth7q\" (UniqueName: \"kubernetes.io/projected/db3da481-af3b-4827-95e5-bdd08e4ff256-kube-api-access-sth7q\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:10.913251 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913238 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:10.913493 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913261 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n92h\" (UniqueName: \"kubernetes.io/projected/ae485556-28dc-4ad6-98e2-7b41753bb66e-kube-api-access-8n92h\") pod \"ingress-canary-fpfnt\" (UID: \"ae485556-28dc-4ad6-98e2-7b41753bb66e\") " pod="openshift-ingress-canary/ingress-canary-fpfnt" Jul 06 19:10:10.913493 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913279 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-bound-sa-token\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:10.913493 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913298 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-image-registry-private-configuration\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:10.913493 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913346 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb102f4e-c53d-459c-aaed-9b20db1d13a6-ca-trust-extracted\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:10.913493 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:10.913370 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-trusted-ca\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.014720 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.014680 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb102f4e-c53d-459c-aaed-9b20db1d13a6-ca-trust-extracted\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.014913 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.014726 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-trusted-ca\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.014913 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.014774 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c7sb\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-kube-api-access-5c7sb\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.014913 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.014791 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae485556-28dc-4ad6-98e2-7b41753bb66e-cert\") pod \"ingress-canary-fpfnt\" (UID: \"ae485556-28dc-4ad6-98e2-7b41753bb66e\") " pod="openshift-ingress-canary/ingress-canary-fpfnt" Jul 06 19:10:11.014913 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.014806 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3da481-af3b-4827-95e5-bdd08e4ff256-tmp-dir\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:11.014913 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.014840 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.015162 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:11.014938 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:10:11.015162 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:11.014956 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84995c7cbf-v2kvf: secret "image-registry-tls" not found Jul 06 19:10:11.015162 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:11.015028 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls podName:cb102f4e-c53d-459c-aaed-9b20db1d13a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:11.515002691 +0000 UTC m=+36.998226259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls") pod "image-registry-84995c7cbf-v2kvf" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6") : secret "image-registry-tls" not found Jul 06 19:10:11.015311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.015199 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb102f4e-c53d-459c-aaed-9b20db1d13a6-ca-trust-extracted\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.015311 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.015268 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db3da481-af3b-4827-95e5-bdd08e4ff256-config-volume\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:11.015410 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.015337 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-installation-pull-secrets\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.015410 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.015397 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-certificates\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.015504 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.015430 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sth7q\" (UniqueName: \"kubernetes.io/projected/db3da481-af3b-4827-95e5-bdd08e4ff256-kube-api-access-sth7q\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:11.015504 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.015456 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db3da481-af3b-4827-95e5-bdd08e4ff256-tmp-dir\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:11.015602 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.015501 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:11.015602 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.015541 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n92h\" (UniqueName: \"kubernetes.io/projected/ae485556-28dc-4ad6-98e2-7b41753bb66e-kube-api-access-8n92h\") pod \"ingress-canary-fpfnt\" (UID: \"ae485556-28dc-4ad6-98e2-7b41753bb66e\") " pod="openshift-ingress-canary/ingress-canary-fpfnt" Jul 06 19:10:11.015602 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.015571 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-bound-sa-token\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.015753 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.015602 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-image-registry-private-configuration\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.016095 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:11.016072 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:10:11.016181 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:11.016134 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls podName:db3da481-af3b-4827-95e5-bdd08e4ff256 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:11.516116548 +0000 UTC m=+36.999340104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls") pod "dns-default-9x7qx" (UID: "db3da481-af3b-4827-95e5-bdd08e4ff256") : secret "dns-default-metrics-tls" not found Jul 06 19:10:11.016181 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.016136 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/db3da481-af3b-4827-95e5-bdd08e4ff256-config-volume\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:11.016181 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.016146 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-certificates\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.016365 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.016341 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-trusted-ca\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.020083 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.020028 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-installation-pull-secrets\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.020083 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.020051 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae485556-28dc-4ad6-98e2-7b41753bb66e-cert\") pod \"ingress-canary-fpfnt\" (UID: \"ae485556-28dc-4ad6-98e2-7b41753bb66e\") " pod="openshift-ingress-canary/ingress-canary-fpfnt" Jul 06 19:10:11.020083 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.020036 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-image-registry-private-configuration\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.030066 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.030033 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sth7q\" (UniqueName: \"kubernetes.io/projected/db3da481-af3b-4827-95e5-bdd08e4ff256-kube-api-access-sth7q\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:11.030179 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.030128 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c7sb\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-kube-api-access-5c7sb\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.033313 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.033263 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-bound-sa-token\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.033518 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.033496 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n92h\" (UniqueName: \"kubernetes.io/projected/ae485556-28dc-4ad6-98e2-7b41753bb66e-kube-api-access-8n92h\") pod \"ingress-canary-fpfnt\" (UID: \"ae485556-28dc-4ad6-98e2-7b41753bb66e\") " pod="openshift-ingress-canary/ingress-canary-fpfnt" Jul 06 19:10:11.141457 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.141423 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fpfnt" Jul 06 19:10:11.284472 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.284209 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fpfnt"] Jul 06 19:10:11.287848 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:11.287820 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae485556_28dc_4ad6_98e2_7b41753bb66e.slice/crio-d55d1504505edfd64b6fe4267c74215c33043028b51bd2e8bc1ef05924e32dbd WatchSource:0}: Error finding container d55d1504505edfd64b6fe4267c74215c33043028b51bd2e8bc1ef05924e32dbd: Status 404 returned error can't find the container with id d55d1504505edfd64b6fe4267c74215c33043028b51bd2e8bc1ef05924e32dbd Jul 06 19:10:11.521078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.521043 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:11.521251 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.521125 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:11.521315 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:11.521260 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:10:11.521315 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:11.521280 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84995c7cbf-v2kvf: secret "image-registry-tls" not found Jul 06 19:10:11.521315 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:11.521300 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:10:11.521417 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:11.521354 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls podName:cb102f4e-c53d-459c-aaed-9b20db1d13a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:12.52132804 +0000 UTC m=+38.004551593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls") pod "image-registry-84995c7cbf-v2kvf" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6") : secret "image-registry-tls" not found Jul 06 19:10:11.521417 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:11.521374 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls podName:db3da481-af3b-4827-95e5-bdd08e4ff256 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:12.521365179 +0000 UTC m=+38.004588734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls") pod "dns-default-9x7qx" (UID: "db3da481-af3b-4827-95e5-bdd08e4ff256") : secret "dns-default-metrics-tls" not found Jul 06 19:10:11.847515 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:11.847435 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fpfnt" event={"ID":"ae485556-28dc-4ad6-98e2-7b41753bb66e","Type":"ContainerStarted","Data":"d55d1504505edfd64b6fe4267c74215c33043028b51bd2e8bc1ef05924e32dbd"} Jul 06 19:10:12.524764 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.524717 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:12.525331 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.524726 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:12.525331 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.524928 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:12.527460 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.527438 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qqjhw\"" Jul 06 19:10:12.527561 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.527487 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:10:12.528847 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.528826 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:10:12.528957 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.528873 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9tqph\"" Jul 06 19:10:12.528957 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.528948 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:10:12.529082 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.529068 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:10:12.530386 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.530365 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:12.530525 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:12.530511 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:10:12.530599 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:12.530549 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:12.530599 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:12.530572 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls podName:db3da481-af3b-4827-95e5-bdd08e4ff256 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:14.53055351 +0000 UTC m=+40.013777090 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls") pod "dns-default-9x7qx" (UID: "db3da481-af3b-4827-95e5-bdd08e4ff256") : secret "dns-default-metrics-tls" not found Jul 06 19:10:12.530704 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:12.530645 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:10:12.530704 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:12.530658 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84995c7cbf-v2kvf: secret "image-registry-tls" not found Jul 06 19:10:12.530704 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:12.530702 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls podName:cb102f4e-c53d-459c-aaed-9b20db1d13a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:14.530686291 +0000 UTC m=+40.013909857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls") pod "image-registry-84995c7cbf-v2kvf" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6") : secret "image-registry-tls" not found Jul 06 19:10:14.549170 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:14.549099 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:14.549170 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:14.549151 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:14.549536 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:14.549244 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:10:14.549536 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:14.549255 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84995c7cbf-v2kvf: secret "image-registry-tls" not found Jul 06 19:10:14.549536 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:14.549252 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:10:14.549536 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:14.549302 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls podName:cb102f4e-c53d-459c-aaed-9b20db1d13a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:18.549289076 +0000 UTC m=+44.032512634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls") pod "image-registry-84995c7cbf-v2kvf" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6") : secret "image-registry-tls" not found Jul 06 19:10:14.549536 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:14.549325 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls podName:db3da481-af3b-4827-95e5-bdd08e4ff256 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:18.549308515 +0000 UTC m=+44.032532066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls") pod "dns-default-9x7qx" (UID: "db3da481-af3b-4827-95e5-bdd08e4ff256") : secret "dns-default-metrics-tls" not found Jul 06 19:10:15.859521 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:15.859432 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fpfnt" event={"ID":"ae485556-28dc-4ad6-98e2-7b41753bb66e","Type":"ContainerStarted","Data":"02ff2803609ba61adda9bd27325bee8a86e81ca1d2c5bdf3caa18a98f2f5e232"} Jul 06 19:10:15.862106 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:15.862080 2554 generic.go:356] "Generic (PLEG): container finished" podID="598c2c71-e257-4445-8f36-710b469c366f" containerID="46083b354798fdf505f3d772caef1c13aaf20a7ed2a5b5f8b363f0fe1c3ca6c2" exitCode=0 Jul 06 19:10:15.862222 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:15.862135 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" event={"ID":"598c2c71-e257-4445-8f36-710b469c366f","Type":"ContainerDied","Data":"46083b354798fdf505f3d772caef1c13aaf20a7ed2a5b5f8b363f0fe1c3ca6c2"} Jul 06 19:10:15.877967 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:15.877915 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fpfnt" podStartSLOduration=1.561392146 podStartE2EDuration="5.877895351s" totalImagesPullingTime="4.316503205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:15.875254156 +0000 UTC m=+41.358477732" watchObservedRunningTime="2026-07-06 19:10:15.877895351 +0000 UTC m=+41.361118924" Jul 06 19:10:16.866821 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:16.866785 2554 generic.go:356] "Generic (PLEG): container finished" podID="598c2c71-e257-4445-8f36-710b469c366f" containerID="d5ce39db9bc7eff2e7bd398b2e369e397620ea527d86c231561cfc90d32852be" exitCode=0 Jul 06 19:10:16.867192 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:16.866874 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" event={"ID":"598c2c71-e257-4445-8f36-710b469c366f","Type":"ContainerDied","Data":"d5ce39db9bc7eff2e7bd398b2e369e397620ea527d86c231561cfc90d32852be"} Jul 06 19:10:17.872966 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:17.872929 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" event={"ID":"598c2c71-e257-4445-8f36-710b469c366f","Type":"ContainerStarted","Data":"e0dc8ea30ab6e63ce226728f388b9ae4a2ff78bb7a9fd30476a01c4b0056176a"} Jul 06 19:10:17.901890 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:17.901834 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xlh9g" podStartSLOduration=8.383921357 podStartE2EDuration="41.90181889s" totalImagesPullingTime="33.517897533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:17.896224027 +0000 UTC m=+43.379447600" watchObservedRunningTime="2026-07-06 19:10:17.90181889 +0000 UTC m=+43.385042462" Jul 06 19:10:18.381250 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.381219 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb"] Jul 06 19:10:18.398592 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.398565 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb"] Jul 06 19:10:18.398718 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.398651 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:18.401389 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.401365 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 19:10:18.401389 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.401376 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 19:10:18.401389 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.401370 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2ntkp\"" Jul 06 19:10:18.485201 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.485170 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf"] Jul 06 19:10:18.500258 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.500228 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4"] Jul 06 19:10:18.500398 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.500341 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.503189 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.503165 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 19:10:18.503336 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.503190 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4fjsb\"" Jul 06 19:10:18.503336 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.503201 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:10:18.503336 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.503174 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 19:10:18.503550 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.503533 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 19:10:18.514259 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.514236 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-58r22"] Jul 06 19:10:18.514374 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.514359 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:18.517546 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.517525 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 19:10:18.517651 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.517553 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:10:18.517651 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.517528 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-k6h8p\"" Jul 06 19:10:18.517933 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.517917 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 19:10:18.536378 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.536355 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.538946 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.538926 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:10:18.539047 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.538935 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 19:10:18.539047 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.538952 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ffbh2\"" Jul 06 19:10:18.539270 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.539251 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 19:10:18.539370 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.539327 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:10:18.539459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.539441 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf"] Jul 06 19:10:18.539545 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.539481 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-58r22"] Jul 06 19:10:18.539545 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.539496 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4"] Jul 06 19:10:18.543960 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.543944 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 19:10:18.581532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.581505 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:18.581654 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.581556 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:18.581654 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.581584 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:18.581654 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.581618 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:18.581778 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:18.581676 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:10:18.581778 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:18.581688 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:10:18.581778 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:18.581693 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84995c7cbf-v2kvf: secret "image-registry-tls" not found Jul 06 19:10:18.581778 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:18.581748 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls podName:db3da481-af3b-4827-95e5-bdd08e4ff256 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:26.581724105 +0000 UTC m=+52.064947657 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls") pod "dns-default-9x7qx" (UID: "db3da481-af3b-4827-95e5-bdd08e4ff256") : secret "dns-default-metrics-tls" not found Jul 06 19:10:18.581778 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:18.581761 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls podName:cb102f4e-c53d-459c-aaed-9b20db1d13a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:26.581755014 +0000 UTC m=+52.064978565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls") pod "image-registry-84995c7cbf-v2kvf" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6") : secret "image-registry-tls" not found Jul 06 19:10:18.682308 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682217 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ca926e-8858-4b62-b157-62833f8411e1-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.682308 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682255 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67tl\" (UniqueName: \"kubernetes.io/projected/11ca926e-8858-4b62-b157-62833f8411e1-kube-api-access-r67tl\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.682308 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682296 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/11ca926e-8858-4b62-b157-62833f8411e1-snapshots\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.682596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682319 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:18.682596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682356 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11ca926e-8858-4b62-b157-62833f8411e1-serving-cert\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.682596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682437 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-mdnbf\" (UID: \"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.682596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682474 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqm52\" (UniqueName: \"kubernetes.io/projected/cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a-kube-api-access-hqm52\") pod \"kube-storage-version-migrator-operator-846bf668f7-mdnbf\" (UID: \"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.682596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682507 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ca926e-8858-4b62-b157-62833f8411e1-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.682596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682540 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-mdnbf\" (UID: \"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.682596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682575 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brv9f\" (UniqueName: \"kubernetes.io/projected/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-kube-api-access-brv9f\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:18.683079 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682605 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:18.683079 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682633 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11ca926e-8858-4b62-b157-62833f8411e1-tmp\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.683079 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.682658 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:18.683079 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:18.682708 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:10:18.683079 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:18.682801 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert podName:af68ec4b-e432-4f9e-92e5-4ec15b9abb3d nodeName:}" failed. No retries permitted until 2026-07-06 19:10:19.182785772 +0000 UTC m=+44.666009326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-w4tgb" (UID: "af68ec4b-e432-4f9e-92e5-4ec15b9abb3d") : secret "networking-console-plugin-cert" not found Jul 06 19:10:18.683475 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.683455 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:18.783913 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.783875 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11ca926e-8858-4b62-b157-62833f8411e1-serving-cert\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.784119 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.783941 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-mdnbf\" (UID: \"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.784119 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.783964 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqm52\" (UniqueName: \"kubernetes.io/projected/cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a-kube-api-access-hqm52\") pod \"kube-storage-version-migrator-operator-846bf668f7-mdnbf\" (UID: \"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.784119 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.783992 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ca926e-8858-4b62-b157-62833f8411e1-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.784119 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784013 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-mdnbf\" (UID: \"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.784119 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784052 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brv9f\" (UniqueName: \"kubernetes.io/projected/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-kube-api-access-brv9f\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:18.784119 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784088 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11ca926e-8858-4b62-b157-62833f8411e1-tmp\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.784399 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784146 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ca926e-8858-4b62-b157-62833f8411e1-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.784399 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784171 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67tl\" (UniqueName: \"kubernetes.io/projected/11ca926e-8858-4b62-b157-62833f8411e1-kube-api-access-r67tl\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.784399 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784218 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/11ca926e-8858-4b62-b157-62833f8411e1-snapshots\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.784399 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784252 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:18.784399 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:18.784357 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:10:18.784627 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:18.784414 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls podName:48599f4d-a4fd-4d49-9bb1-d0e211f5c18b nodeName:}" failed. No retries permitted until 2026-07-06 19:10:19.284397791 +0000 UTC m=+44.767621360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wc2p4" (UID: "48599f4d-a4fd-4d49-9bb1-d0e211f5c18b") : secret "samples-operator-tls" not found Jul 06 19:10:18.784692 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784628 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-mdnbf\" (UID: \"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.784809 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784785 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11ca926e-8858-4b62-b157-62833f8411e1-tmp\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.784857 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784831 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ca926e-8858-4b62-b157-62833f8411e1-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.784993 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.784972 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ca926e-8858-4b62-b157-62833f8411e1-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.785100 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.785083 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/11ca926e-8858-4b62-b157-62833f8411e1-snapshots\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.787224 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.787199 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-mdnbf\" (UID: \"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.787321 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.787233 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/11ca926e-8858-4b62-b157-62833f8411e1-serving-cert\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.792786 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.792759 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqm52\" (UniqueName: \"kubernetes.io/projected/cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a-kube-api-access-hqm52\") pod \"kube-storage-version-migrator-operator-846bf668f7-mdnbf\" (UID: \"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.792987 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.792972 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67tl\" (UniqueName: \"kubernetes.io/projected/11ca926e-8858-4b62-b157-62833f8411e1-kube-api-access-r67tl\") pod \"insights-operator-5c847dcfc4-58r22\" (UID: \"11ca926e-8858-4b62-b157-62833f8411e1\") " pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.793693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.793676 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brv9f\" (UniqueName: \"kubernetes.io/projected/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-kube-api-access-brv9f\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:18.809544 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.809525 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" Jul 06 19:10:18.844682 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.844663 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-58r22" Jul 06 19:10:18.991584 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:18.991550 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf"] Jul 06 19:10:18.996284 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:18.996250 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0881a9_cdc6_4fd5_928c_1d5c9030cc4a.slice/crio-b724cf00f226bda2fd2f806f804564e24e53f7abf4ce5aff6e912e3b6904f4fd WatchSource:0}: Error finding container b724cf00f226bda2fd2f806f804564e24e53f7abf4ce5aff6e912e3b6904f4fd: Status 404 returned error can't find the container with id b724cf00f226bda2fd2f806f804564e24e53f7abf4ce5aff6e912e3b6904f4fd Jul 06 19:10:19.004229 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:19.004204 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-58r22"] Jul 06 19:10:19.006516 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:19.006489 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11ca926e_8858_4b62_b157_62833f8411e1.slice/crio-ace6628a676e1b71bdbd9e436a318b4ed8c74c5d979e1a85ad0b3fc2df81eb07 WatchSource:0}: Error finding container ace6628a676e1b71bdbd9e436a318b4ed8c74c5d979e1a85ad0b3fc2df81eb07: Status 404 returned error can't find the container with id ace6628a676e1b71bdbd9e436a318b4ed8c74c5d979e1a85ad0b3fc2df81eb07 Jul 06 19:10:19.188180 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:19.188137 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:19.188366 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:19.188284 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:10:19.188366 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:19.188350 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert podName:af68ec4b-e432-4f9e-92e5-4ec15b9abb3d nodeName:}" failed. No retries permitted until 2026-07-06 19:10:20.188335276 +0000 UTC m=+45.671558831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-w4tgb" (UID: "af68ec4b-e432-4f9e-92e5-4ec15b9abb3d") : secret "networking-console-plugin-cert" not found Jul 06 19:10:19.289173 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:19.289090 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:19.289334 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:19.289250 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:10:19.289334 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:19.289328 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls podName:48599f4d-a4fd-4d49-9bb1-d0e211f5c18b nodeName:}" failed. No retries permitted until 2026-07-06 19:10:20.289308163 +0000 UTC m=+45.772531732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wc2p4" (UID: "48599f4d-a4fd-4d49-9bb1-d0e211f5c18b") : secret "samples-operator-tls" not found Jul 06 19:10:19.879986 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:19.879950 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-58r22" event={"ID":"11ca926e-8858-4b62-b157-62833f8411e1","Type":"ContainerStarted","Data":"ace6628a676e1b71bdbd9e436a318b4ed8c74c5d979e1a85ad0b3fc2df81eb07"} Jul 06 19:10:19.881146 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:19.881118 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" event={"ID":"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a","Type":"ContainerStarted","Data":"b724cf00f226bda2fd2f806f804564e24e53f7abf4ce5aff6e912e3b6904f4fd"} Jul 06 19:10:20.195431 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:20.195339 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:20.195888 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:20.195512 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:10:20.195888 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:20.195600 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert podName:af68ec4b-e432-4f9e-92e5-4ec15b9abb3d nodeName:}" failed. No retries permitted until 2026-07-06 19:10:22.195578283 +0000 UTC m=+47.678801846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-w4tgb" (UID: "af68ec4b-e432-4f9e-92e5-4ec15b9abb3d") : secret "networking-console-plugin-cert" not found Jul 06 19:10:20.296402 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:20.296367 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:20.296543 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:20.296511 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:10:20.296598 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:20.296577 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls podName:48599f4d-a4fd-4d49-9bb1-d0e211f5c18b nodeName:}" failed. No retries permitted until 2026-07-06 19:10:22.296561992 +0000 UTC m=+47.779785546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wc2p4" (UID: "48599f4d-a4fd-4d49-9bb1-d0e211f5c18b") : secret "samples-operator-tls" not found Jul 06 19:10:22.211256 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:22.211231 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:22.211698 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:22.211366 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:10:22.211698 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:22.211423 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert podName:af68ec4b-e432-4f9e-92e5-4ec15b9abb3d nodeName:}" failed. No retries permitted until 2026-07-06 19:10:26.21140839 +0000 UTC m=+51.694631956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-w4tgb" (UID: "af68ec4b-e432-4f9e-92e5-4ec15b9abb3d") : secret "networking-console-plugin-cert" not found Jul 06 19:10:22.312566 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:22.312486 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:22.312724 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:22.312627 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:10:22.312724 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:22.312683 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls podName:48599f4d-a4fd-4d49-9bb1-d0e211f5c18b nodeName:}" failed. No retries permitted until 2026-07-06 19:10:26.312670312 +0000 UTC m=+51.795893866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wc2p4" (UID: "48599f4d-a4fd-4d49-9bb1-d0e211f5c18b") : secret "samples-operator-tls" not found Jul 06 19:10:22.890765 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:22.890712 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-58r22" event={"ID":"11ca926e-8858-4b62-b157-62833f8411e1","Type":"ContainerStarted","Data":"bc77abe18b11732eb956b9a94564b88e0bde4ef547de7e16d718ff651df88473"} Jul 06 19:10:22.892399 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:22.892372 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" event={"ID":"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a","Type":"ContainerStarted","Data":"d9ac0fcee06c77df737fbd16e79fcd547f2150945de43a8db7fbbcbf00eda781"} Jul 06 19:10:22.908090 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:22.908041 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-58r22" podStartSLOduration=1.8497105980000002 podStartE2EDuration="4.908027286s" totalImagesPullingTime="3.058316688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:22.907455061 +0000 UTC m=+48.390678635" watchObservedRunningTime="2026-07-06 19:10:22.908027286 +0000 UTC m=+48.391250837" Jul 06 19:10:22.924254 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:22.924205 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" podStartSLOduration=1.850190795 podStartE2EDuration="4.924186693s" totalImagesPullingTime="3.073995898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:22.923765894 +0000 UTC m=+48.406989465" watchObservedRunningTime="2026-07-06 19:10:22.924186693 +0000 UTC m=+48.407410267" Jul 06 19:10:24.431751 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.431702 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jljqf"] Jul 06 19:10:24.481957 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.481913 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jljqf"] Jul 06 19:10:24.481957 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.481937 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.485288 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.485263 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:10:24.485406 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.485263 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:10:24.485406 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.485269 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jdr5p\"" Jul 06 19:10:24.531461 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.531437 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60469931-e60f-4ff2-958c-147f23e5c775-crio-socket\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.531604 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.531469 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60469931-e60f-4ff2-958c-147f23e5c775-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.531604 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.531520 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.531604 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.531555 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60469931-e60f-4ff2-958c-147f23e5c775-data-volume\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.531698 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.531663 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zjf\" (UniqueName: \"kubernetes.io/projected/60469931-e60f-4ff2-958c-147f23e5c775-kube-api-access-g2zjf\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.633053 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.633023 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60469931-e60f-4ff2-958c-147f23e5c775-crio-socket\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.633053 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.633059 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60469931-e60f-4ff2-958c-147f23e5c775-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.633269 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.633101 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.633269 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.633137 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60469931-e60f-4ff2-958c-147f23e5c775-data-volume\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.633269 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:24.633167 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:10:24.633269 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.633183 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zjf\" (UniqueName: \"kubernetes.io/projected/60469931-e60f-4ff2-958c-147f23e5c775-kube-api-access-g2zjf\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.633269 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:24.633239 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60469931-e60f-4ff2-958c-147f23e5c775-kube-rbac-proxy-cm podName:60469931-e60f-4ff2-958c-147f23e5c775 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:25.133219209 +0000 UTC m=+50.616442777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60469931-e60f-4ff2-958c-147f23e5c775-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jljqf" (UID: "60469931-e60f-4ff2-958c-147f23e5c775") : configmap "kube-rbac-proxy" not found Jul 06 19:10:24.633269 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.633251 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/60469931-e60f-4ff2-958c-147f23e5c775-crio-socket\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.633500 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:24.633279 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:10:24.633500 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:24.633336 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls podName:60469931-e60f-4ff2-958c-147f23e5c775 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:25.133318912 +0000 UTC m=+50.616542463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jljqf" (UID: "60469931-e60f-4ff2-958c-147f23e5c775") : secret "insights-runtime-extractor-tls" not found Jul 06 19:10:24.633500 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.633491 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/60469931-e60f-4ff2-958c-147f23e5c775-data-volume\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:24.641325 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:24.641306 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zjf\" (UniqueName: \"kubernetes.io/projected/60469931-e60f-4ff2-958c-147f23e5c775-kube-api-access-g2zjf\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:25.136610 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:25.136565 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60469931-e60f-4ff2-958c-147f23e5c775-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:25.136845 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:25.136632 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:25.136845 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:25.136695 2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:10:25.136845 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:25.136746 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:10:25.136845 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:25.136796 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60469931-e60f-4ff2-958c-147f23e5c775-kube-rbac-proxy-cm podName:60469931-e60f-4ff2-958c-147f23e5c775 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:26.136774035 +0000 UTC m=+51.619997587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/60469931-e60f-4ff2-958c-147f23e5c775-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jljqf" (UID: "60469931-e60f-4ff2-958c-147f23e5c775") : configmap "kube-rbac-proxy" not found Jul 06 19:10:25.136845 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:25.136817 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls podName:60469931-e60f-4ff2-958c-147f23e5c775 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:26.136808561 +0000 UTC m=+51.620032115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jljqf" (UID: "60469931-e60f-4ff2-958c-147f23e5c775") : secret "insights-runtime-extractor-tls" not found Jul 06 19:10:26.144452 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:26.144410 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:26.144958 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:26.144554 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60469931-e60f-4ff2-958c-147f23e5c775-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:26.144958 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.144568 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:10:26.144958 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.144633 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls podName:60469931-e60f-4ff2-958c-147f23e5c775 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:28.144615675 +0000 UTC m=+53.627839493 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jljqf" (UID: "60469931-e60f-4ff2-958c-147f23e5c775") : secret "insights-runtime-extractor-tls" not found Jul 06 19:10:26.145169 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:26.145149 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/60469931-e60f-4ff2-958c-147f23e5c775-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:26.244900 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:26.244865 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:26.245051 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.245030 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:10:26.245111 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.245099 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert podName:af68ec4b-e432-4f9e-92e5-4ec15b9abb3d nodeName:}" failed. No retries permitted until 2026-07-06 19:10:34.245084331 +0000 UTC m=+59.728307882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-w4tgb" (UID: "af68ec4b-e432-4f9e-92e5-4ec15b9abb3d") : secret "networking-console-plugin-cert" not found Jul 06 19:10:26.346127 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:26.346091 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:26.346323 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.346261 2554 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 19:10:26.346397 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.346341 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls podName:48599f4d-a4fd-4d49-9bb1-d0e211f5c18b nodeName:}" failed. No retries permitted until 2026-07-06 19:10:34.346320458 +0000 UTC m=+59.829544010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-wc2p4" (UID: "48599f4d-a4fd-4d49-9bb1-d0e211f5c18b") : secret "samples-operator-tls" not found Jul 06 19:10:26.649583 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:26.649538 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:26.649875 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:26.649657 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:26.649875 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.649657 2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:10:26.649875 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.649703 2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:10:26.649875 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.649718 2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84995c7cbf-v2kvf: secret "image-registry-tls" not found Jul 06 19:10:26.649875 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.649790 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls podName:db3da481-af3b-4827-95e5-bdd08e4ff256 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:42.649769439 +0000 UTC m=+68.132993011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls") pod "dns-default-9x7qx" (UID: "db3da481-af3b-4827-95e5-bdd08e4ff256") : secret "dns-default-metrics-tls" not found Jul 06 19:10:26.649875 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:26.649809 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls podName:cb102f4e-c53d-459c-aaed-9b20db1d13a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:42.649800339 +0000 UTC m=+68.133023890 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls") pod "image-registry-84995c7cbf-v2kvf" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6") : secret "image-registry-tls" not found Jul 06 19:10:27.212632 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:27.212608 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-56v7k_c1c2a219-c8c6-4937-952e-630b3eb7de5a/dns-node-resolver/0.log" Jul 06 19:10:28.162166 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:28.162126 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:28.162408 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:28.162277 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:10:28.162408 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:28.162342 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls podName:60469931-e60f-4ff2-958c-147f23e5c775 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:32.162327895 +0000 UTC m=+57.645551451 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jljqf" (UID: "60469931-e60f-4ff2-958c-147f23e5c775") : secret "insights-runtime-extractor-tls" not found Jul 06 19:10:28.412498 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:28.412423 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xqhsh_eaebfdaf-a012-490a-8e17-b8346cbf6748/node-ca/0.log" Jul 06 19:10:28.812991 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:28.812921 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fpfnt_ae485556-28dc-4ad6-98e2-7b41753bb66e/serve-healthcheck-canary/0.log" Jul 06 19:10:29.813460 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:29.813428 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-mdnbf_cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a/kube-storage-version-migrator-operator/0.log" Jul 06 19:10:32.195820 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:32.195782 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:32.196247 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:32.195927 2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:10:32.196247 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:32.196004 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls podName:60469931-e60f-4ff2-958c-147f23e5c775 nodeName:}" failed. No retries permitted until 2026-07-06 19:10:40.195988812 +0000 UTC m=+65.679212364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jljqf" (UID: "60469931-e60f-4ff2-958c-147f23e5c775") : secret "insights-runtime-extractor-tls" not found Jul 06 19:10:34.312968 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:34.312916 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:34.313376 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:34.313065 2554 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 19:10:34.313376 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:10:34.313132 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert podName:af68ec4b-e432-4f9e-92e5-4ec15b9abb3d nodeName:}" failed. No retries permitted until 2026-07-06 19:10:50.313115561 +0000 UTC m=+75.796339117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-w4tgb" (UID: "af68ec4b-e432-4f9e-92e5-4ec15b9abb3d") : secret "networking-console-plugin-cert" not found Jul 06 19:10:34.414146 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:34.414107 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:34.416470 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:34.416446 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/48599f4d-a4fd-4d49-9bb1-d0e211f5c18b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-wc2p4\" (UID: \"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:34.423312 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:34.423292 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" Jul 06 19:10:34.544719 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:34.544690 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4"] Jul 06 19:10:34.922690 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:34.922656 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" event={"ID":"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b","Type":"ContainerStarted","Data":"88a78ab0f90ae518cca9a2d95a4de8b0e2459de115a736f5c056e05f20d3b2b5"} Jul 06 19:10:35.836511 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:35.836481 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bfz6c" Jul 06 19:10:37.931528 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:37.931492 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" event={"ID":"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b","Type":"ContainerStarted","Data":"428dd88f5e666431fe635a455c2c4d1880213da92e2dec23de618f091458667e"} Jul 06 19:10:37.931976 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:37.931548 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" event={"ID":"48599f4d-a4fd-4d49-9bb1-d0e211f5c18b","Type":"ContainerStarted","Data":"b19bf41d0330b1af45f83891c5c060b06ed64e820d1ef4bb249b998341a9d62f"} Jul 06 19:10:37.957748 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:37.957691 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-wc2p4" podStartSLOduration=17.64907431 podStartE2EDuration="19.957672627s" totalImagesPullingTime="2.308598317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:37.957515509 +0000 UTC m=+63.440739083" watchObservedRunningTime="2026-07-06 19:10:37.957672627 +0000 UTC m=+63.440896201" Jul 06 19:10:40.257551 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:40.257501 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:40.259742 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:40.259713 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/60469931-e60f-4ff2-958c-147f23e5c775-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jljqf\" (UID: \"60469931-e60f-4ff2-958c-147f23e5c775\") " pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:40.395642 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:40.395611 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jdr5p\"" Jul 06 19:10:40.402184 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:40.402167 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jljqf" Jul 06 19:10:40.527957 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:40.527934 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jljqf"] Jul 06 19:10:40.528080 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:40.528031 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60469931_e60f_4ff2_958c_147f23e5c775.slice/crio-a3c0b756dafd4266faa666d11f1dddb21668b42b4578e278d763e8a406acbe2e WatchSource:0}: Error finding container a3c0b756dafd4266faa666d11f1dddb21668b42b4578e278d763e8a406acbe2e: Status 404 returned error can't find the container with id a3c0b756dafd4266faa666d11f1dddb21668b42b4578e278d763e8a406acbe2e Jul 06 19:10:40.941026 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:40.940988 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jljqf" event={"ID":"60469931-e60f-4ff2-958c-147f23e5c775","Type":"ContainerStarted","Data":"a3c0b756dafd4266faa666d11f1dddb21668b42b4578e278d763e8a406acbe2e"} Jul 06 19:10:42.174466 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.174425 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:42.177444 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.177417 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:10:42.187758 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.187725 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/423a3192-16a1-4e66-94de-9d88c359469c-original-pull-secret\") pod \"global-pull-secret-syncer-ncc6d\" (UID: \"423a3192-16a1-4e66-94de-9d88c359469c\") " pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:42.244160 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.244123 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncc6d" Jul 06 19:10:42.274973 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.274946 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:42.278320 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.278115 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:10:42.288310 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.288257 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee50f9b8-fc16-4951-a486-88d460005a0c-metrics-certs\") pod \"network-metrics-daemon-hlrkz\" (UID: \"ee50f9b8-fc16-4951-a486-88d460005a0c\") " pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:42.376431 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.376389 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j6h8\" (UniqueName: \"kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8\") pod \"network-check-target-g5gwx\" (UID: \"9b975cfc-6a57-4352-8833-61170b28d14b\") " pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:42.379881 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.379852 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:10:42.384810 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.384754 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ncc6d"] Jul 06 19:10:42.388182 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:42.388146 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423a3192_16a1_4e66_94de_9d88c359469c.slice/crio-d3f2d454440c9812f3214f719a1e87314ab9ff1a93f01d9755e210141fa6aed5 WatchSource:0}: Error finding container d3f2d454440c9812f3214f719a1e87314ab9ff1a93f01d9755e210141fa6aed5: Status 404 returned error can't find the container with id d3f2d454440c9812f3214f719a1e87314ab9ff1a93f01d9755e210141fa6aed5 Jul 06 19:10:42.389338 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.389315 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:10:42.399207 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.399185 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j6h8\" (UniqueName: \"kubernetes.io/projected/9b975cfc-6a57-4352-8833-61170b28d14b-kube-api-access-4j6h8\") pod \"network-check-target-g5gwx\" (UID: \"9b975cfc-6a57-4352-8833-61170b28d14b\") " pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:42.540369 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.540286 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qqjhw\"" Jul 06 19:10:42.547634 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.547611 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hlrkz" Jul 06 19:10:42.552771 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.552726 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9tqph\"" Jul 06 19:10:42.559974 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.559945 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:42.680257 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.680222 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:42.680430 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.680279 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:42.682959 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.682930 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/db3da481-af3b-4827-95e5-bdd08e4ff256-metrics-tls\") pod \"dns-default-9x7qx\" (UID: \"db3da481-af3b-4827-95e5-bdd08e4ff256\") " pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:42.683073 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.682937 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") pod \"image-registry-84995c7cbf-v2kvf\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:42.687849 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.687811 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hlrkz"] Jul 06 19:10:42.706632 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.706602 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5gwx"] Jul 06 19:10:42.915704 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.915671 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5jctw\"" Jul 06 19:10:42.923148 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.923124 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:42.927330 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.927304 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2k95g\"" Jul 06 19:10:42.935426 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.935408 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:42.950954 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:42.950924 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ncc6d" event={"ID":"423a3192-16a1-4e66-94de-9d88c359469c","Type":"ContainerStarted","Data":"d3f2d454440c9812f3214f719a1e87314ab9ff1a93f01d9755e210141fa6aed5"} Jul 06 19:10:43.834688 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:43.834645 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee50f9b8_fc16_4951_a486_88d460005a0c.slice/crio-15ff9b77a451ab771c982c26950f0dd2dd3405efd170e752643794c93af9c57a WatchSource:0}: Error finding container 15ff9b77a451ab771c982c26950f0dd2dd3405efd170e752643794c93af9c57a: Status 404 returned error can't find the container with id 15ff9b77a451ab771c982c26950f0dd2dd3405efd170e752643794c93af9c57a Jul 06 19:10:43.835620 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:43.835535 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b975cfc_6a57_4352_8833_61170b28d14b.slice/crio-9dafd051f53c4e4df966d4c078c65ff74ffe24bedfd67664cdd1704dee02a177 WatchSource:0}: Error finding container 9dafd051f53c4e4df966d4c078c65ff74ffe24bedfd67664cdd1704dee02a177: Status 404 returned error can't find the container with id 9dafd051f53c4e4df966d4c078c65ff74ffe24bedfd67664cdd1704dee02a177 Jul 06 19:10:43.956431 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:43.956390 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hlrkz" event={"ID":"ee50f9b8-fc16-4951-a486-88d460005a0c","Type":"ContainerStarted","Data":"15ff9b77a451ab771c982c26950f0dd2dd3405efd170e752643794c93af9c57a"} Jul 06 19:10:43.957658 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:43.957631 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5gwx" event={"ID":"9b975cfc-6a57-4352-8833-61170b28d14b","Type":"ContainerStarted","Data":"9dafd051f53c4e4df966d4c078c65ff74ffe24bedfd67664cdd1704dee02a177"} Jul 06 19:10:44.976828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:44.976794 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9x7qx"] Jul 06 19:10:44.980829 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:44.980795 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3da481_af3b_4827_95e5_bdd08e4ff256.slice/crio-4b0e40ce444c579625d4cdd599e65c587c30e43e78e7680f6001c3781763767f WatchSource:0}: Error finding container 4b0e40ce444c579625d4cdd599e65c587c30e43e78e7680f6001c3781763767f: Status 404 returned error can't find the container with id 4b0e40ce444c579625d4cdd599e65c587c30e43e78e7680f6001c3781763767f Jul 06 19:10:45.013825 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:45.013767 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84995c7cbf-v2kvf"] Jul 06 19:10:45.018008 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:45.017976 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb102f4e_c53d_459c_aaed_9b20db1d13a6.slice/crio-43618059c8f4c665e243d872f568e73f7795927b90ff5fc5cbe97260dda67785 WatchSource:0}: Error finding container 43618059c8f4c665e243d872f568e73f7795927b90ff5fc5cbe97260dda67785: Status 404 returned error can't find the container with id 43618059c8f4c665e243d872f568e73f7795927b90ff5fc5cbe97260dda67785 Jul 06 19:10:45.974291 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:45.973914 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hlrkz" event={"ID":"ee50f9b8-fc16-4951-a486-88d460005a0c","Type":"ContainerStarted","Data":"9f996123de0d25291f7763009dc3c3470aa46ef291f0dbe1d53261ee6b542ded"} Jul 06 19:10:45.974291 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:45.973974 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hlrkz" event={"ID":"ee50f9b8-fc16-4951-a486-88d460005a0c","Type":"ContainerStarted","Data":"4f0d1159a79437f9af6efc7979a935c87c806870c287e80e4c54110546abae06"} Jul 06 19:10:45.976339 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:45.976225 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" event={"ID":"cb102f4e-c53d-459c-aaed-9b20db1d13a6","Type":"ContainerStarted","Data":"a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14"} Jul 06 19:10:45.976339 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:45.976265 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" event={"ID":"cb102f4e-c53d-459c-aaed-9b20db1d13a6","Type":"ContainerStarted","Data":"43618059c8f4c665e243d872f568e73f7795927b90ff5fc5cbe97260dda67785"} Jul 06 19:10:45.976538 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:45.976436 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:10:45.979147 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:45.979120 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jljqf" event={"ID":"60469931-e60f-4ff2-958c-147f23e5c775","Type":"ContainerStarted","Data":"a48c6d1d9fc809621782f5650d8505b40924a4b14a68e6ab0619d3df10964f92"} Jul 06 19:10:45.979670 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:45.979224 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jljqf" event={"ID":"60469931-e60f-4ff2-958c-147f23e5c775","Type":"ContainerStarted","Data":"2548e678b313173078c6c8b9eb9cc097038e3e1fdce5391e9a249ca7d87b68f6"} Jul 06 19:10:45.981241 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:45.981202 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x7qx" event={"ID":"db3da481-af3b-4827-95e5-bdd08e4ff256","Type":"ContainerStarted","Data":"4b0e40ce444c579625d4cdd599e65c587c30e43e78e7680f6001c3781763767f"} Jul 06 19:10:45.991848 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:45.991798 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hlrkz" podStartSLOduration=67.250895998 podStartE2EDuration="1m8.991781502s" totalImagesPullingTime="1.740885504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:45.991244274 +0000 UTC m=+71.474467849" watchObservedRunningTime="2026-07-06 19:10:45.991781502 +0000 UTC m=+71.475005077" Jul 06 19:10:46.011847 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:46.011794 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" podStartSLOduration=71.01177344 podStartE2EDuration="1m11.01177344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:46.01104985 +0000 UTC m=+71.494273423" watchObservedRunningTime="2026-07-06 19:10:46.01177344 +0000 UTC m=+71.494997014" Jul 06 19:10:50.344252 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:50.344211 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:50.346660 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:50.346637 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af68ec4b-e432-4f9e-92e5-4ec15b9abb3d-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-w4tgb\" (UID: \"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:50.509493 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:50.509459 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2ntkp\"" Jul 06 19:10:50.516897 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:50.516868 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" Jul 06 19:10:51.300394 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.300368 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb"] Jul 06 19:10:51.308026 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:51.307998 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf68ec4b_e432_4f9e_92e5_4ec15b9abb3d.slice/crio-2668bfbe0947d0b095d051be3a0d56b4a5e8f80168f85a7d751ce326f606d7ea WatchSource:0}: Error finding container 2668bfbe0947d0b095d051be3a0d56b4a5e8f80168f85a7d751ce326f606d7ea: Status 404 returned error can't find the container with id 2668bfbe0947d0b095d051be3a0d56b4a5e8f80168f85a7d751ce326f606d7ea Jul 06 19:10:51.589548 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.589511 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb"] Jul 06 19:10:51.603638 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.603602 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.604256 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.604224 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb"] Jul 06 19:10:51.607216 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.607196 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:10:51.608648 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.608626 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 19:10:51.608776 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.608666 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:10:51.608776 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.608702 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:10:51.655427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.655390 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8955b6fe-b16d-4e82-8f97-c084ed217e42-klusterlet-config\") pod \"klusterlet-addon-workmgr-68957dc5d6-dm7bb\" (UID: \"8955b6fe-b16d-4e82-8f97-c084ed217e42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.655598 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.655432 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmhn6\" (UniqueName: \"kubernetes.io/projected/8955b6fe-b16d-4e82-8f97-c084ed217e42-kube-api-access-tmhn6\") pod \"klusterlet-addon-workmgr-68957dc5d6-dm7bb\" (UID: \"8955b6fe-b16d-4e82-8f97-c084ed217e42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.655598 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.655467 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8955b6fe-b16d-4e82-8f97-c084ed217e42-tmp\") pod \"klusterlet-addon-workmgr-68957dc5d6-dm7bb\" (UID: \"8955b6fe-b16d-4e82-8f97-c084ed217e42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.692804 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.692776 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-ktkpf"] Jul 06 19:10:51.706786 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.706766 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-ktkpf" Jul 06 19:10:51.708887 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.708862 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-ktkpf"] Jul 06 19:10:51.711699 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.711675 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:10:51.711815 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.711797 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:10:51.711955 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.711941 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hccxf\"" Jul 06 19:10:51.756246 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.756217 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8955b6fe-b16d-4e82-8f97-c084ed217e42-klusterlet-config\") pod \"klusterlet-addon-workmgr-68957dc5d6-dm7bb\" (UID: \"8955b6fe-b16d-4e82-8f97-c084ed217e42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.756246 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.756249 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmhn6\" (UniqueName: \"kubernetes.io/projected/8955b6fe-b16d-4e82-8f97-c084ed217e42-kube-api-access-tmhn6\") pod \"klusterlet-addon-workmgr-68957dc5d6-dm7bb\" (UID: \"8955b6fe-b16d-4e82-8f97-c084ed217e42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.756423 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.756278 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8955b6fe-b16d-4e82-8f97-c084ed217e42-tmp\") pod \"klusterlet-addon-workmgr-68957dc5d6-dm7bb\" (UID: \"8955b6fe-b16d-4e82-8f97-c084ed217e42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.756683 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.756663 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8955b6fe-b16d-4e82-8f97-c084ed217e42-tmp\") pod \"klusterlet-addon-workmgr-68957dc5d6-dm7bb\" (UID: \"8955b6fe-b16d-4e82-8f97-c084ed217e42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.758860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.758843 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/8955b6fe-b16d-4e82-8f97-c084ed217e42-klusterlet-config\") pod \"klusterlet-addon-workmgr-68957dc5d6-dm7bb\" (UID: \"8955b6fe-b16d-4e82-8f97-c084ed217e42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.766363 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.766321 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmhn6\" (UniqueName: \"kubernetes.io/projected/8955b6fe-b16d-4e82-8f97-c084ed217e42-kube-api-access-tmhn6\") pod \"klusterlet-addon-workmgr-68957dc5d6-dm7bb\" (UID: \"8955b6fe-b16d-4e82-8f97-c084ed217e42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.856680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.856594 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whkjk\" (UniqueName: \"kubernetes.io/projected/fabc8a56-c8b0-4def-9e61-5b5430ac4538-kube-api-access-whkjk\") pod \"downloads-685bfb5f64-ktkpf\" (UID: \"fabc8a56-c8b0-4def-9e61-5b5430ac4538\") " pod="openshift-console/downloads-685bfb5f64-ktkpf" Jul 06 19:10:51.914338 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.914307 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:51.957561 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.957531 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whkjk\" (UniqueName: \"kubernetes.io/projected/fabc8a56-c8b0-4def-9e61-5b5430ac4538-kube-api-access-whkjk\") pod \"downloads-685bfb5f64-ktkpf\" (UID: \"fabc8a56-c8b0-4def-9e61-5b5430ac4538\") " pod="openshift-console/downloads-685bfb5f64-ktkpf" Jul 06 19:10:51.970474 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:51.970444 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whkjk\" (UniqueName: \"kubernetes.io/projected/fabc8a56-c8b0-4def-9e61-5b5430ac4538-kube-api-access-whkjk\") pod \"downloads-685bfb5f64-ktkpf\" (UID: \"fabc8a56-c8b0-4def-9e61-5b5430ac4538\") " pod="openshift-console/downloads-685bfb5f64-ktkpf" Jul 06 19:10:52.005650 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.005615 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jljqf" event={"ID":"60469931-e60f-4ff2-958c-147f23e5c775","Type":"ContainerStarted","Data":"d43d5adcd79d394a123562262331d52dc75715eae8b866b0bdfe103cc4b85f21"} Jul 06 19:10:52.006896 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.006868 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" event={"ID":"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d","Type":"ContainerStarted","Data":"2668bfbe0947d0b095d051be3a0d56b4a5e8f80168f85a7d751ce326f606d7ea"} Jul 06 19:10:52.008347 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.008322 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5gwx" event={"ID":"9b975cfc-6a57-4352-8833-61170b28d14b","Type":"ContainerStarted","Data":"7044bcd3e5bcc1abeeecdfd4453e768fdafded2673ac639e3aca4d45df008d4c"} Jul 06 19:10:52.008480 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.008387 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:10:52.011451 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.011389 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x7qx" event={"ID":"db3da481-af3b-4827-95e5-bdd08e4ff256","Type":"ContainerStarted","Data":"0116c54cf25f47718bc1cfe9272c482a4a8135bea129aa2148906ce8ad9e9f8b"} Jul 06 19:10:52.011571 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.011455 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9x7qx" Jul 06 19:10:52.011571 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.011479 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9x7qx" event={"ID":"db3da481-af3b-4827-95e5-bdd08e4ff256","Type":"ContainerStarted","Data":"f38617cdcbca14c98f2061d4237348aa97fc53a5fbbfa8272ecb0989d8a91d48"} Jul 06 19:10:52.013262 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.013220 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ncc6d" event={"ID":"423a3192-16a1-4e66-94de-9d88c359469c","Type":"ContainerStarted","Data":"6cdd0096e005666a33f1300c4a941352f37ad1094bfde6ccfd50b4beff2a0446"} Jul 06 19:10:52.016823 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.016799 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-ktkpf" Jul 06 19:10:52.054953 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.054893 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g5gwx" podStartSLOduration=67.735397047 podStartE2EDuration="1m15.054856474s" totalImagesPullingTime="7.319459427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:52.054855934 +0000 UTC m=+77.538079508" watchObservedRunningTime="2026-07-06 19:10:52.054856474 +0000 UTC m=+77.538080049" Jul 06 19:10:52.055195 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.055164 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jljqf" podStartSLOduration=17.603902307 podStartE2EDuration="28.055153106s" totalImagesPullingTime="10.451250799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:10:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:52.024772574 +0000 UTC m=+77.507996150" watchObservedRunningTime="2026-07-06 19:10:52.055153106 +0000 UTC m=+77.538376678" Jul 06 19:10:52.074871 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.074838 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb"] Jul 06 19:10:52.082763 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.082636 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9x7qx" podStartSLOduration=35.909689557 podStartE2EDuration="42.082616258s" totalImagesPullingTime="6.172926701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:52.082078529 +0000 UTC m=+77.565302102" watchObservedRunningTime="2026-07-06 19:10:52.082616258 +0000 UTC m=+77.565839831" Jul 06 19:10:52.102898 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.102844 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ncc6d" podStartSLOduration=66.324657979 podStartE2EDuration="1m15.102823879s" totalImagesPullingTime="8.7781659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:52.101969126 +0000 UTC m=+77.585192711" watchObservedRunningTime="2026-07-06 19:10:52.102823879 +0000 UTC m=+77.586047452" Jul 06 19:10:52.164173 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.164142 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-ktkpf"] Jul 06 19:10:52.166910 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:52.166886 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfabc8a56_c8b0_4def_9e61_5b5430ac4538.slice/crio-f9c7cc438edef5124dc7a80c237c3520c5bdfaf7678c975741d1fec8d4e46d30 WatchSource:0}: Error finding container f9c7cc438edef5124dc7a80c237c3520c5bdfaf7678c975741d1fec8d4e46d30: Status 404 returned error can't find the container with id f9c7cc438edef5124dc7a80c237c3520c5bdfaf7678c975741d1fec8d4e46d30 Jul 06 19:10:52.303889 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.303862 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b74c8cf7b-6f6x5"] Jul 06 19:10:52.347860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.347676 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b74c8cf7b-6f6x5"] Jul 06 19:10:52.347860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.347821 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.350974 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.350759 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-76gqn\"" Jul 06 19:10:52.350974 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.350823 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 19:10:52.351176 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.351070 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 19:10:52.351176 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.351131 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 19:10:52.351300 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.351281 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 19:10:52.351364 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.351343 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 19:10:52.462964 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.462931 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-service-ca\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.462964 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.462965 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-serving-cert\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.463200 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.462993 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrh2\" (UniqueName: \"kubernetes.io/projected/051be0d1-2510-4728-b6f6-c3d7381d16da-kube-api-access-8vrh2\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.463200 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.463021 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-console-config\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.463200 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.463045 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-oauth-serving-cert\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.463200 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.463061 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-oauth-config\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.564224 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.564187 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-service-ca\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.564224 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.564225 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-serving-cert\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.564505 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.564280 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrh2\" (UniqueName: \"kubernetes.io/projected/051be0d1-2510-4728-b6f6-c3d7381d16da-kube-api-access-8vrh2\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.564505 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.564315 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-console-config\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.564505 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.564338 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-oauth-serving-cert\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.564505 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.564364 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-oauth-config\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.565019 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.564997 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-service-ca\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.565219 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.565074 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-console-config\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.565219 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.565107 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-oauth-serving-cert\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.566684 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.566658 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-oauth-config\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.566816 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.566747 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-serving-cert\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.573082 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.573051 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrh2\" (UniqueName: \"kubernetes.io/projected/051be0d1-2510-4728-b6f6-c3d7381d16da-kube-api-access-8vrh2\") pod \"console-6b74c8cf7b-6f6x5\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.669783 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.669711 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:10:52.804670 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:52.804647 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b74c8cf7b-6f6x5"] Jul 06 19:10:52.810027 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:52.809993 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051be0d1_2510_4728_b6f6_c3d7381d16da.slice/crio-f323462f1d7cfd9f377a54ea75db5de7b3a9f659ca367c9e94e849d2788812e9 WatchSource:0}: Error finding container f323462f1d7cfd9f377a54ea75db5de7b3a9f659ca367c9e94e849d2788812e9: Status 404 returned error can't find the container with id f323462f1d7cfd9f377a54ea75db5de7b3a9f659ca367c9e94e849d2788812e9 Jul 06 19:10:53.019813 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:53.019703 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" event={"ID":"af68ec4b-e432-4f9e-92e5-4ec15b9abb3d","Type":"ContainerStarted","Data":"ba328f38dcba02082fcc21225ea0bc5d2f5b2b0873e7b7ab66c0009fa12fcc69"} Jul 06 19:10:53.021352 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:53.021320 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b74c8cf7b-6f6x5" event={"ID":"051be0d1-2510-4728-b6f6-c3d7381d16da","Type":"ContainerStarted","Data":"f323462f1d7cfd9f377a54ea75db5de7b3a9f659ca367c9e94e849d2788812e9"} Jul 06 19:10:53.022768 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:53.022720 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-ktkpf" event={"ID":"fabc8a56-c8b0-4def-9e61-5b5430ac4538","Type":"ContainerStarted","Data":"f9c7cc438edef5124dc7a80c237c3520c5bdfaf7678c975741d1fec8d4e46d30"} Jul 06 19:10:53.024079 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:53.024044 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" event={"ID":"8955b6fe-b16d-4e82-8f97-c084ed217e42","Type":"ContainerStarted","Data":"a5eb8e05ea35ada58bdd45dea10d4c5a746e8ee69f947fd87be5c89f820e652b"} Jul 06 19:10:53.036892 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:53.036722 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-w4tgb" podStartSLOduration=33.58722909 podStartE2EDuration="35.036704892s" totalImagesPullingTime="1.449475802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:53.03521155 +0000 UTC m=+78.518435126" watchObservedRunningTime="2026-07-06 19:10:53.036704892 +0000 UTC m=+78.519928465" Jul 06 19:10:54.497210 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.497177 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c9f48d9d8-ccxpt"] Jul 06 19:10:54.505585 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.505554 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.513343 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.513301 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c9f48d9d8-ccxpt"] Jul 06 19:10:54.517044 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.516930 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 19:10:54.582514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.582175 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdf6\" (UniqueName: \"kubernetes.io/projected/dcf309f5-cd5f-44df-848b-9a3c21381aa8-kube-api-access-7mdf6\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.582514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.582219 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-serving-cert\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.582514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.582292 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-oauth-config\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.582514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.582316 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-service-ca\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.582514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.582340 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-oauth-serving-cert\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.582514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.582370 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-config\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.582514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.582403 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-trusted-ca-bundle\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.682774 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.682724 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-oauth-config\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.682774 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.682779 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-service-ca\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.682976 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.682803 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-oauth-serving-cert\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.682976 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.682831 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-config\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.682976 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.682862 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-trusted-ca-bundle\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.683099 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.683015 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdf6\" (UniqueName: \"kubernetes.io/projected/dcf309f5-cd5f-44df-848b-9a3c21381aa8-kube-api-access-7mdf6\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.683099 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.683064 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-serving-cert\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.683746 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.683702 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-config\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.683879 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.683823 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-service-ca\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.684057 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.683930 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-oauth-serving-cert\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.684467 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.684419 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-trusted-ca-bundle\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.685906 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.685866 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-oauth-config\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.686006 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.685907 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-serving-cert\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.692251 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.692214 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdf6\" (UniqueName: \"kubernetes.io/projected/dcf309f5-cd5f-44df-848b-9a3c21381aa8-kube-api-access-7mdf6\") pod \"console-6c9f48d9d8-ccxpt\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.820422 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.820374 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:10:54.978425 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:54.978342 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c9f48d9d8-ccxpt"] Jul 06 19:10:54.982650 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:54.982620 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf309f5_cd5f_44df_848b_9a3c21381aa8.slice/crio-79b653b3bd7987b861b43cc45b756180f136410150a3776aced77b7742c9e5ab WatchSource:0}: Error finding container 79b653b3bd7987b861b43cc45b756180f136410150a3776aced77b7742c9e5ab: Status 404 returned error can't find the container with id 79b653b3bd7987b861b43cc45b756180f136410150a3776aced77b7742c9e5ab Jul 06 19:10:55.035453 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.035408 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9f48d9d8-ccxpt" event={"ID":"dcf309f5-cd5f-44df-848b-9a3c21381aa8","Type":"ContainerStarted","Data":"79b653b3bd7987b861b43cc45b756180f136410150a3776aced77b7742c9e5ab"} Jul 06 19:10:55.294411 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.294325 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67"] Jul 06 19:10:55.297005 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.296982 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67" Jul 06 19:10:55.299516 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.299491 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 19:10:55.299757 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.299724 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qlm25\"" Jul 06 19:10:55.304180 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.304143 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67"] Jul 06 19:10:55.389214 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.389090 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/699f4628-62ca-48d0-b7fa-4ac5b289b9a8-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vng67\" (UID: \"699f4628-62ca-48d0-b7fa-4ac5b289b9a8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67" Jul 06 19:10:55.491652 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.491610 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/699f4628-62ca-48d0-b7fa-4ac5b289b9a8-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vng67\" (UID: \"699f4628-62ca-48d0-b7fa-4ac5b289b9a8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67" Jul 06 19:10:55.498460 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.498404 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/699f4628-62ca-48d0-b7fa-4ac5b289b9a8-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-vng67\" (UID: \"699f4628-62ca-48d0-b7fa-4ac5b289b9a8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67" Jul 06 19:10:55.609629 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.609592 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67" Jul 06 19:10:55.767002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:55.766448 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67"] Jul 06 19:10:57.154350 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:10:57.154235 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699f4628_62ca_48d0_b7fa_4ac5b289b9a8.slice/crio-c343a2dbf871c5b9aedf5d0dc7524e8284aec3886b36f7682f8d37d634ed07ac WatchSource:0}: Error finding container c343a2dbf871c5b9aedf5d0dc7524e8284aec3886b36f7682f8d37d634ed07ac: Status 404 returned error can't find the container with id c343a2dbf871c5b9aedf5d0dc7524e8284aec3886b36f7682f8d37d634ed07ac Jul 06 19:10:58.049689 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:58.049638 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67" event={"ID":"699f4628-62ca-48d0-b7fa-4ac5b289b9a8","Type":"ContainerStarted","Data":"c343a2dbf871c5b9aedf5d0dc7524e8284aec3886b36f7682f8d37d634ed07ac"} Jul 06 19:10:59.055001 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:59.054716 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b74c8cf7b-6f6x5" event={"ID":"051be0d1-2510-4728-b6f6-c3d7381d16da","Type":"ContainerStarted","Data":"e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d"} Jul 06 19:10:59.056421 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:59.056384 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" event={"ID":"8955b6fe-b16d-4e82-8f97-c084ed217e42","Type":"ContainerStarted","Data":"aa136fa655b0255f5ca8d2f9a9824d09e94c60da179f9d3322f1cfd9aecd1094"} Jul 06 19:10:59.056586 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:59.056567 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:59.058333 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:59.058308 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9f48d9d8-ccxpt" event={"ID":"dcf309f5-cd5f-44df-848b-9a3c21381aa8","Type":"ContainerStarted","Data":"dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce"} Jul 06 19:10:59.058516 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:59.058498 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" Jul 06 19:10:59.075641 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:59.075590 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b74c8cf7b-6f6x5" podStartSLOduration=1.599360457 podStartE2EDuration="7.075575724s" totalImagesPullingTime="5.476215267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:59.074002043 +0000 UTC m=+84.557225616" watchObservedRunningTime="2026-07-06 19:10:59.075575724 +0000 UTC m=+84.558799288" Jul 06 19:10:59.093619 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:10:59.093572 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68957dc5d6-dm7bb" podStartSLOduration=1.883093533 podStartE2EDuration="8.093555945s" totalImagesPullingTime="6.210462412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:59.092855055 +0000 UTC m=+84.576078628" watchObservedRunningTime="2026-07-06 19:10:59.093555945 +0000 UTC m=+84.576779517" Jul 06 19:11:00.063460 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:00.063417 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67" event={"ID":"699f4628-62ca-48d0-b7fa-4ac5b289b9a8","Type":"ContainerStarted","Data":"dc89f4c53bc7cb8135076ef0297b08a6b824687583fd883b878762b8cad89bd7"} Jul 06 19:11:00.082356 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:00.082307 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67" podStartSLOduration=3.192872574 podStartE2EDuration="5.082293404s" totalImagesPullingTime="1.88942083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:00.08161703 +0000 UTC m=+85.564840601" watchObservedRunningTime="2026-07-06 19:11:00.082293404 +0000 UTC m=+85.565516977" Jul 06 19:11:00.082532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:00.082514 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c9f48d9d8-ccxpt" podStartSLOduration=2.772982948 podStartE2EDuration="6.082509356s" totalImagesPullingTime="3.309526408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:59.118047159 +0000 UTC m=+84.601270732" watchObservedRunningTime="2026-07-06 19:11:00.082509356 +0000 UTC m=+85.565732928" Jul 06 19:11:01.068124 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:01.068068 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67" Jul 06 19:11:01.073298 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:01.073247 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-vng67" Jul 06 19:11:02.026126 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:02.026093 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9x7qx" Jul 06 19:11:02.669983 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:02.669952 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:11:02.670398 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:02.670227 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:11:02.671459 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:02.671432 2554 patch_prober.go:28] interesting pod/console-6b74c8cf7b-6f6x5 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" start-of-body= Jul 06 19:11:02.671594 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:02.671482 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6b74c8cf7b-6f6x5" podUID="051be0d1-2510-4728-b6f6-c3d7381d16da" containerName="console" probeResult="failure" output="Get \"https://10.133.0.16:8443/health\": dial tcp 10.133.0.16:8443: connect: connection refused" Jul 06 19:11:02.928144 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:02.928050 2554 patch_prober.go:28] interesting pod/image-registry-84995c7cbf-v2kvf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 19:11:02.928144 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:02.928113 2554 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" podUID="cb102f4e-c53d-459c-aaed-9b20db1d13a6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:11:04.820995 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:04.820954 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:11:04.821363 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:04.821021 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:11:04.825452 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:04.825430 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:11:05.089649 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:05.089577 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:11:05.145506 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:05.145474 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b74c8cf7b-6f6x5"] Jul 06 19:11:06.727082 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.727038 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4"] Jul 06 19:11:06.730741 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.730709 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.733969 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.733939 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 19:11:06.734173 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.733955 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 19:11:06.734340 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.734319 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:11:06.735619 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.735581 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:11:06.735619 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.735607 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-c7s7z\"" Jul 06 19:11:06.735774 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.735656 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:11:06.743356 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.743339 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4"] Jul 06 19:11:06.753948 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.753925 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm"] Jul 06 19:11:06.756861 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.756843 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.760114 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.760091 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 19:11:06.760362 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.760343 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 19:11:06.760669 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.760650 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qwjmj\"" Jul 06 19:11:06.760774 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.760690 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 19:11:06.771879 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.771861 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-874h4"] Jul 06 19:11:06.775596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.775565 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm"] Jul 06 19:11:06.775706 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.775693 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.778475 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.778455 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:11:06.778585 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.778511 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:11:06.778661 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.778644 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rphr7\"" Jul 06 19:11:06.778720 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.778702 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:11:06.890585 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890551 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-wtmp\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.890585 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890589 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4d54b830-a51c-4979-b181-4c47e5988b07-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.890874 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890608 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvt27\" (UniqueName: \"kubernetes.io/projected/4d54b830-a51c-4979-b181-4c47e5988b07-kube-api-access-hvt27\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.890874 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890625 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d21596dd-12ac-4c85-aeff-ac0b60181817-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.890874 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890644 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4d54b830-a51c-4979-b181-4c47e5988b07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.890874 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890667 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d21596dd-12ac-4c85-aeff-ac0b60181817-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.890874 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890689 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d54b830-a51c-4979-b181-4c47e5988b07-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.890874 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890710 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d54b830-a51c-4979-b181-4c47e5988b07-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.890874 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890795 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxgl\" (UniqueName: \"kubernetes.io/projected/d21596dd-12ac-4c85-aeff-ac0b60181817-kube-api-access-wmxgl\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.890874 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890833 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f039c33b-9c32-4295-a96a-4526f3de292d-root\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.890874 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890861 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-tls\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.891255 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890888 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.891255 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890933 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f039c33b-9c32-4295-a96a-4526f3de292d-sys\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.891255 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.890959 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d21596dd-12ac-4c85-aeff-ac0b60181817-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.891255 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.891005 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d54b830-a51c-4979-b181-4c47e5988b07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.891255 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.891044 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-textfile\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.891255 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.891071 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f039c33b-9c32-4295-a96a-4526f3de292d-metrics-client-ca\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.891255 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.891097 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlnht\" (UniqueName: \"kubernetes.io/projected/f039c33b-9c32-4295-a96a-4526f3de292d-kube-api-access-nlnht\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.891255 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.891128 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-accelerators-collector-config\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.990866 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.990795 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:11:06.991476 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991460 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxgl\" (UniqueName: \"kubernetes.io/projected/d21596dd-12ac-4c85-aeff-ac0b60181817-kube-api-access-wmxgl\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.991532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991485 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f039c33b-9c32-4295-a96a-4526f3de292d-root\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.991532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991502 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-tls\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.991532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991521 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.991671 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991583 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f039c33b-9c32-4295-a96a-4526f3de292d-root\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.991671 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:11:06.991592 2554 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 19:11:06.991671 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991639 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f039c33b-9c32-4295-a96a-4526f3de292d-sys\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.991858 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:11:06.991681 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-tls podName:f039c33b-9c32-4295-a96a-4526f3de292d nodeName:}" failed. No retries permitted until 2026-07-06 19:11:07.491659756 +0000 UTC m=+92.974883333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-tls") pod "node-exporter-874h4" (UID: "f039c33b-9c32-4295-a96a-4526f3de292d") : secret "node-exporter-tls" not found Jul 06 19:11:06.991858 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991690 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f039c33b-9c32-4295-a96a-4526f3de292d-sys\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.991858 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991717 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d21596dd-12ac-4c85-aeff-ac0b60181817-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.991858 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991787 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d54b830-a51c-4979-b181-4c47e5988b07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.991858 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991836 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-textfile\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.992115 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991863 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f039c33b-9c32-4295-a96a-4526f3de292d-metrics-client-ca\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.992115 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991888 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlnht\" (UniqueName: \"kubernetes.io/projected/f039c33b-9c32-4295-a96a-4526f3de292d-kube-api-access-nlnht\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.992115 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991922 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-accelerators-collector-config\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.992115 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.991970 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-wtmp\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.992115 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992009 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4d54b830-a51c-4979-b181-4c47e5988b07-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.992115 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992034 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvt27\" (UniqueName: \"kubernetes.io/projected/4d54b830-a51c-4979-b181-4c47e5988b07-kube-api-access-hvt27\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.992115 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992061 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d21596dd-12ac-4c85-aeff-ac0b60181817-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.992115 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992093 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4d54b830-a51c-4979-b181-4c47e5988b07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.992490 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992125 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d21596dd-12ac-4c85-aeff-ac0b60181817-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.992490 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992164 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d54b830-a51c-4979-b181-4c47e5988b07-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.992490 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992202 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-textfile\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.992490 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992209 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d54b830-a51c-4979-b181-4c47e5988b07-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.992685 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992562 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-wtmp\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.992685 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992630 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f039c33b-9c32-4295-a96a-4526f3de292d-metrics-client-ca\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.993038 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992861 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-accelerators-collector-config\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.993038 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.992948 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d54b830-a51c-4979-b181-4c47e5988b07-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.993141 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.993045 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4d54b830-a51c-4979-b181-4c47e5988b07-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.993274 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.993255 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4d54b830-a51c-4979-b181-4c47e5988b07-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.993531 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.993509 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d21596dd-12ac-4c85-aeff-ac0b60181817-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.994461 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.994434 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:06.995122 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.995098 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d54b830-a51c-4979-b181-4c47e5988b07-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.995265 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.995203 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d21596dd-12ac-4c85-aeff-ac0b60181817-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:06.995390 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.995371 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d54b830-a51c-4979-b181-4c47e5988b07-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:06.995513 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:06.995495 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d21596dd-12ac-4c85-aeff-ac0b60181817-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:07.001849 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:07.001822 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlnht\" (UniqueName: \"kubernetes.io/projected/f039c33b-9c32-4295-a96a-4526f3de292d-kube-api-access-nlnht\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:07.002187 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:07.002171 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxgl\" (UniqueName: \"kubernetes.io/projected/d21596dd-12ac-4c85-aeff-ac0b60181817-kube-api-access-wmxgl\") pod \"openshift-state-metrics-54cf96dfcf-m6vd4\" (UID: \"d21596dd-12ac-4c85-aeff-ac0b60181817\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:07.002597 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:07.002581 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvt27\" (UniqueName: \"kubernetes.io/projected/4d54b830-a51c-4979-b181-4c47e5988b07-kube-api-access-hvt27\") pod \"kube-state-metrics-7c9dbb8b7d-6d5sm\" (UID: \"4d54b830-a51c-4979-b181-4c47e5988b07\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:07.041547 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:07.041519 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" Jul 06 19:11:07.066811 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:07.066778 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" Jul 06 19:11:07.194056 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:07.194025 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4"] Jul 06 19:11:07.195501 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:11:07.195472 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21596dd_12ac_4c85_aeff_ac0b60181817.slice/crio-c08a694f53b3ae98c84a582bac76f406b2b408897766c47f1bd202fa0de2386c WatchSource:0}: Error finding container c08a694f53b3ae98c84a582bac76f406b2b408897766c47f1bd202fa0de2386c: Status 404 returned error can't find the container with id c08a694f53b3ae98c84a582bac76f406b2b408897766c47f1bd202fa0de2386c Jul 06 19:11:07.223683 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:07.223550 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm"] Jul 06 19:11:07.226301 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:11:07.226270 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d54b830_a51c_4979_b181_4c47e5988b07.slice/crio-6c237e716f40495b8d28a06429ccbcb6cbf6c5c9ba5111587c8992a6748e6117 WatchSource:0}: Error finding container 6c237e716f40495b8d28a06429ccbcb6cbf6c5c9ba5111587c8992a6748e6117: Status 404 returned error can't find the container with id 6c237e716f40495b8d28a06429ccbcb6cbf6c5c9ba5111587c8992a6748e6117 Jul 06 19:11:07.496952 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:07.496805 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-tls\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:07.499405 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:07.499380 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f039c33b-9c32-4295-a96a-4526f3de292d-node-exporter-tls\") pod \"node-exporter-874h4\" (UID: \"f039c33b-9c32-4295-a96a-4526f3de292d\") " pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:07.686135 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:07.686101 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-874h4" Jul 06 19:11:07.696759 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:11:07.696715 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf039c33b_9c32_4295_a96a_4526f3de292d.slice/crio-64854fbaafed733111ede9b5c47ff813e05ef6745c8475f7581f7fb23e6c1f69 WatchSource:0}: Error finding container 64854fbaafed733111ede9b5c47ff813e05ef6745c8475f7581f7fb23e6c1f69: Status 404 returned error can't find the container with id 64854fbaafed733111ede9b5c47ff813e05ef6745c8475f7581f7fb23e6c1f69 Jul 06 19:11:08.101994 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:08.101950 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" event={"ID":"4d54b830-a51c-4979-b181-4c47e5988b07","Type":"ContainerStarted","Data":"6c237e716f40495b8d28a06429ccbcb6cbf6c5c9ba5111587c8992a6748e6117"} Jul 06 19:11:08.103563 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:08.103497 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-874h4" event={"ID":"f039c33b-9c32-4295-a96a-4526f3de292d","Type":"ContainerStarted","Data":"64854fbaafed733111ede9b5c47ff813e05ef6745c8475f7581f7fb23e6c1f69"} Jul 06 19:11:08.106022 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:08.105986 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" event={"ID":"d21596dd-12ac-4c85-aeff-ac0b60181817","Type":"ContainerStarted","Data":"5e6f096875e7bc07c512aa51a7446ca1f2bb130fddcbee400aa0172de910bd97"} Jul 06 19:11:08.106163 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:08.106053 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" event={"ID":"d21596dd-12ac-4c85-aeff-ac0b60181817","Type":"ContainerStarted","Data":"b98d3dd5494cb83e783a472ba4b72cfd796ee8dde199a03093148ddf8bb0a541"} Jul 06 19:11:08.106163 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:08.106083 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" event={"ID":"d21596dd-12ac-4c85-aeff-ac0b60181817","Type":"ContainerStarted","Data":"c08a694f53b3ae98c84a582bac76f406b2b408897766c47f1bd202fa0de2386c"} Jul 06 19:11:10.116524 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:10.116485 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" event={"ID":"4d54b830-a51c-4979-b181-4c47e5988b07","Type":"ContainerStarted","Data":"ad892ce9ec63c74337b052d163614153485d9ef3a99845fd2ce5cfbb9e682380"} Jul 06 19:11:10.116972 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:10.116548 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" event={"ID":"4d54b830-a51c-4979-b181-4c47e5988b07","Type":"ContainerStarted","Data":"abbccd147483f9aef6478524b5d10bdf995370762961d857eb4fc838509f2843"} Jul 06 19:11:10.116972 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:10.116570 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" event={"ID":"4d54b830-a51c-4979-b181-4c47e5988b07","Type":"ContainerStarted","Data":"dc54fe153c324529ddba5f170626c07f11faa76ae8c384cdda55e4578ecb11b0"} Jul 06 19:11:10.118169 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:10.118147 2554 generic.go:356] "Generic (PLEG): container finished" podID="f039c33b-9c32-4295-a96a-4526f3de292d" containerID="8df932be71a33b0b3bca789588832eb1f8c30e5ed2c04f3fa53ad56de887cc93" exitCode=0 Jul 06 19:11:10.118259 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:10.118237 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-874h4" event={"ID":"f039c33b-9c32-4295-a96a-4526f3de292d","Type":"ContainerDied","Data":"8df932be71a33b0b3bca789588832eb1f8c30e5ed2c04f3fa53ad56de887cc93"} Jul 06 19:11:10.120348 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:10.120323 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" event={"ID":"d21596dd-12ac-4c85-aeff-ac0b60181817","Type":"ContainerStarted","Data":"6a244e76efc8d123bcfad8e57796bd160ef251a5458e55f1a0de207ccd066503"} Jul 06 19:11:10.149649 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:10.149604 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6d5sm" podStartSLOduration=2.231448276 podStartE2EDuration="4.149588265s" totalImagesPullingTime="1.918139989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:10.148884445 +0000 UTC m=+95.632108018" watchObservedRunningTime="2026-07-06 19:11:10.149588265 +0000 UTC m=+95.632811853" Jul 06 19:11:10.194524 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:10.194480 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-m6vd4" podStartSLOduration=2.366589757 podStartE2EDuration="4.194463518s" totalImagesPullingTime="1.827873761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:10.192246004 +0000 UTC m=+95.675469604" watchObservedRunningTime="2026-07-06 19:11:10.194463518 +0000 UTC m=+95.677687069" Jul 06 19:11:11.126024 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.125980 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-874h4" event={"ID":"f039c33b-9c32-4295-a96a-4526f3de292d","Type":"ContainerStarted","Data":"d15a0b85e3153f9cd0ec3d7c1994c4569ed3443f2fa8bc2fed081c267ac18905"} Jul 06 19:11:11.126440 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.126048 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-874h4" event={"ID":"f039c33b-9c32-4295-a96a-4526f3de292d","Type":"ContainerStarted","Data":"4668ee81084a545c246290e060994af797820d0b78793ef0d1c4b25c568a091b"} Jul 06 19:11:11.142400 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.142374 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-55495fc5d5-fpvk6"] Jul 06 19:11:11.144725 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.144708 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.148866 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.148835 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9hqpmvojuf68b\"" Jul 06 19:11:11.148975 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.148928 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 19:11:11.148975 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.148928 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9ts2v\"" Jul 06 19:11:11.149065 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.149016 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 19:11:11.149065 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.149059 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 19:11:11.149206 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.149190 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 19:11:11.149347 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.149309 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-874h4" podStartSLOduration=3.603962724 podStartE2EDuration="5.149295902s" totalImagesPullingTime="1.448832833s" totalInitContainerRuntime="96.500345ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:11.147664523 +0000 UTC m=+96.630888098" watchObservedRunningTime="2026-07-06 19:11:11.149295902 +0000 UTC m=+96.632519477" Jul 06 19:11:11.159820 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.159799 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55495fc5d5-fpvk6"] Jul 06 19:11:11.227292 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.227253 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-secret-metrics-server-client-certs\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.227292 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.227289 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-secret-metrics-server-tls\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.227537 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.227305 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-metrics-server-audit-profiles\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.227537 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.227379 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-client-ca-bundle\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.227537 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.227417 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.227537 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.227489 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-audit-log\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.227721 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.227569 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xngsp\" (UniqueName: \"kubernetes.io/projected/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-kube-api-access-xngsp\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.328680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.328640 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xngsp\" (UniqueName: \"kubernetes.io/projected/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-kube-api-access-xngsp\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.328904 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.328722 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-secret-metrics-server-client-certs\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.328904 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.328765 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-secret-metrics-server-tls\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.328904 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.328780 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-metrics-server-audit-profiles\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.328904 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.328801 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-client-ca-bundle\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.328904 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.328827 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.328904 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.328852 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-audit-log\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.329394 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.329365 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-audit-log\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.329583 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.329559 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.329949 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.329927 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-metrics-server-audit-profiles\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.331346 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.331327 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-client-ca-bundle\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.331419 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.331354 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-secret-metrics-server-tls\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.331419 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.331392 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-secret-metrics-server-client-certs\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.336644 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.336618 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xngsp\" (UniqueName: \"kubernetes.io/projected/b6e1ac1c-7d0a-4306-ad97-2639ab10ed81-kube-api-access-xngsp\") pod \"metrics-server-55495fc5d5-fpvk6\" (UID: \"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81\") " pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.454167 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.454074 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:11.495045 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.495009 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn"] Jul 06 19:11:11.498144 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.498120 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:11.501076 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.500971 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 19:11:11.501076 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.501053 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cnftc\"" Jul 06 19:11:11.508026 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.507514 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn"] Jul 06 19:11:11.531033 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.530934 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbk2\" (UniqueName: \"kubernetes.io/projected/e7cffb20-d0f2-4323-bc35-e5695a8339f0-kube-api-access-bsbk2\") pod \"monitoring-plugin-d5dcff7cf-6bgpn\" (UID: \"e7cffb20-d0f2-4323-bc35-e5695a8339f0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:11.531033 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.530982 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e7cffb20-d0f2-4323-bc35-e5695a8339f0-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6bgpn\" (UID: \"e7cffb20-d0f2-4323-bc35-e5695a8339f0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:11.598685 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.598658 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55495fc5d5-fpvk6"] Jul 06 19:11:11.632052 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.632016 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbk2\" (UniqueName: \"kubernetes.io/projected/e7cffb20-d0f2-4323-bc35-e5695a8339f0-kube-api-access-bsbk2\") pod \"monitoring-plugin-d5dcff7cf-6bgpn\" (UID: \"e7cffb20-d0f2-4323-bc35-e5695a8339f0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:11.632191 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.632062 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e7cffb20-d0f2-4323-bc35-e5695a8339f0-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6bgpn\" (UID: \"e7cffb20-d0f2-4323-bc35-e5695a8339f0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:11.632273 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:11:11.632247 2554 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 19:11:11.632340 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:11:11.632329 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7cffb20-d0f2-4323-bc35-e5695a8339f0-monitoring-plugin-cert podName:e7cffb20-d0f2-4323-bc35-e5695a8339f0 nodeName:}" failed. No retries permitted until 2026-07-06 19:11:12.132309482 +0000 UTC m=+97.615533053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/e7cffb20-d0f2-4323-bc35-e5695a8339f0-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-6bgpn" (UID: "e7cffb20-d0f2-4323-bc35-e5695a8339f0") : secret "monitoring-plugin-cert" not found Jul 06 19:11:11.641419 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:11.641387 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbk2\" (UniqueName: \"kubernetes.io/projected/e7cffb20-d0f2-4323-bc35-e5695a8339f0-kube-api-access-bsbk2\") pod \"monitoring-plugin-d5dcff7cf-6bgpn\" (UID: \"e7cffb20-d0f2-4323-bc35-e5695a8339f0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:12.137626 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.137588 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e7cffb20-d0f2-4323-bc35-e5695a8339f0-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6bgpn\" (UID: \"e7cffb20-d0f2-4323-bc35-e5695a8339f0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:12.140201 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.140176 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e7cffb20-d0f2-4323-bc35-e5695a8339f0-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-6bgpn\" (UID: \"e7cffb20-d0f2-4323-bc35-e5695a8339f0\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:12.286529 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.286488 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b68f7cb55-hstjc"] Jul 06 19:11:12.290541 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.290517 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.304073 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.304044 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b68f7cb55-hstjc"] Jul 06 19:11:12.339044 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.338934 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-serving-cert\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.339044 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.339002 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlwf\" (UniqueName: \"kubernetes.io/projected/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-kube-api-access-5dlwf\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.339303 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.339243 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-service-ca\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.339460 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.339421 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-config\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.339615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.339480 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-oauth-config\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.339615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.339572 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-oauth-serving-cert\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.339901 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.339615 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-trusted-ca-bundle\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.415131 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.415048 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:12.440791 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.440760 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-serving-cert\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.440933 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.440796 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlwf\" (UniqueName: \"kubernetes.io/projected/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-kube-api-access-5dlwf\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.440933 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.440827 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-service-ca\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.440933 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.440889 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-config\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.440933 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.440921 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-oauth-config\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.441118 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.440969 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-oauth-serving-cert\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.441118 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.440994 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-trusted-ca-bundle\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.441747 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.441698 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-service-ca\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.441863 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.441767 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-oauth-serving-cert\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.441863 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.441824 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-config\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.441975 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.441931 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-trusted-ca-bundle\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.443552 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.443528 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-serving-cert\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.443661 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.443638 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-oauth-config\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.450076 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.450048 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlwf\" (UniqueName: \"kubernetes.io/projected/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-kube-api-access-5dlwf\") pod \"console-7b68f7cb55-hstjc\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.602726 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.602687 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:12.893288 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.893254 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:11:12.899404 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.899372 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.902360 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.902339 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 19:11:12.902485 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.902459 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 19:11:12.902596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.902495 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bdqu806cq33df\"" Jul 06 19:11:12.902596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.902435 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 19:11:12.902596 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.902366 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 19:11:12.902783 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.902569 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 19:11:12.902783 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.902686 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 19:11:12.902783 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.902703 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 19:11:12.904064 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.903900 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 19:11:12.904064 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.903975 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 19:11:12.904064 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.903987 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 19:11:12.904268 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.904227 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 19:11:12.904268 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.904235 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dknq4\"" Jul 06 19:11:12.904710 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.904691 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 19:11:12.907759 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.907610 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 19:11:12.910059 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.910039 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 19:11:12.913987 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.913967 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:11:12.945830 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.945805 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-web-config\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.945950 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.945839 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.945950 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.945865 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.945950 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.945890 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.945950 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.945930 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946126 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946017 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-config\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946126 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946041 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79aa8a2b-7643-4a49-b603-57fb45202849-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946126 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946073 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946126 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946108 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946275 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946134 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946275 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946206 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rsrn\" (UniqueName: \"kubernetes.io/projected/79aa8a2b-7643-4a49-b603-57fb45202849-kube-api-access-9rsrn\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946275 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946245 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946396 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946298 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946396 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946340 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946396 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946371 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946530 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946400 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946530 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946470 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946530 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946496 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946530 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946523 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79aa8a2b-7643-4a49-b603-57fb45202849-config-out\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:12.946757 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:12.946600 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047411 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047375 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047586 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047444 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rsrn\" (UniqueName: \"kubernetes.io/projected/79aa8a2b-7643-4a49-b603-57fb45202849-kube-api-access-9rsrn\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047586 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047485 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047586 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047508 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047586 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047551 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047577 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047610 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047643 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047665 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047692 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79aa8a2b-7643-4a49-b603-57fb45202849-config-out\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047752 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047788 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-web-config\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.047828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047823 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048194 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047851 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048194 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047896 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048194 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047920 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048194 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047973 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-config\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048194 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.047999 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79aa8a2b-7643-4a49-b603-57fb45202849-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048194 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.048029 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048194 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.048054 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048194 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.048150 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048750 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.048714 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048750 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.048721 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.048886 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.048757 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.049304 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.048971 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.049719 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.049693 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.050552 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.050507 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.051033 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.050701 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.051536 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.051511 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/79aa8a2b-7643-4a49-b603-57fb45202849-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.051793 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.051744 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.053116 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.053075 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-web-config\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.053116 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.053108 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.053275 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.053256 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/79aa8a2b-7643-4a49-b603-57fb45202849-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.053348 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.053324 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.053468 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.053448 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.054020 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.053988 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/79aa8a2b-7643-4a49-b603-57fb45202849-config-out\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.054264 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.054238 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.054721 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.054693 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.054833 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.054823 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/79aa8a2b-7643-4a49-b603-57fb45202849-config\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.056185 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.056165 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rsrn\" (UniqueName: \"kubernetes.io/projected/79aa8a2b-7643-4a49-b603-57fb45202849-kube-api-access-9rsrn\") pod \"prometheus-k8s-0\" (UID: \"79aa8a2b-7643-4a49-b603-57fb45202849\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.211497 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.211400 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:13.649615 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:13.649581 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84995c7cbf-v2kvf"] Jul 06 19:11:18.380231 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:11:18.380192 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e1ac1c_7d0a_4306_ad97_2639ab10ed81.slice/crio-bc5e3530ae034878e903e16b66eaa1aef96522c982a81e1d5ebb9bef070755c4 WatchSource:0}: Error finding container bc5e3530ae034878e903e16b66eaa1aef96522c982a81e1d5ebb9bef070755c4: Status 404 returned error can't find the container with id bc5e3530ae034878e903e16b66eaa1aef96522c982a81e1d5ebb9bef070755c4 Jul 06 19:11:18.545608 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:18.545581 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn"] Jul 06 19:11:18.547669 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:11:18.547641 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cffb20_d0f2_4323_bc35_e5695a8339f0.slice/crio-6028537d33681244d9298d0392deb8f968b8abc656da9dc68ae07bd84be594d1 WatchSource:0}: Error finding container 6028537d33681244d9298d0392deb8f968b8abc656da9dc68ae07bd84be594d1: Status 404 returned error can't find the container with id 6028537d33681244d9298d0392deb8f968b8abc656da9dc68ae07bd84be594d1 Jul 06 19:11:18.767819 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:18.767726 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b68f7cb55-hstjc"] Jul 06 19:11:18.769918 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:11:18.769887 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d85f11_b3e9_48bb_8dfd_eb9c9490b255.slice/crio-fc1517e904af2935a496fc9157b8c7ad5c447c75f1a1d22a12068c4568b688cb WatchSource:0}: Error finding container fc1517e904af2935a496fc9157b8c7ad5c447c75f1a1d22a12068c4568b688cb: Status 404 returned error can't find the container with id fc1517e904af2935a496fc9157b8c7ad5c447c75f1a1d22a12068c4568b688cb Jul 06 19:11:18.778565 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:18.778533 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 19:11:18.780934 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:11:18.780910 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79aa8a2b_7643_4a49_b603_57fb45202849.slice/crio-c3954d6f7fd2506e6afabb0cf4887371e43d09408153d7d7e9f9778cc76a8a58 WatchSource:0}: Error finding container c3954d6f7fd2506e6afabb0cf4887371e43d09408153d7d7e9f9778cc76a8a58: Status 404 returned error can't find the container with id c3954d6f7fd2506e6afabb0cf4887371e43d09408153d7d7e9f9778cc76a8a58 Jul 06 19:11:19.162753 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.162705 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-ktkpf" event={"ID":"fabc8a56-c8b0-4def-9e61-5b5430ac4538","Type":"ContainerStarted","Data":"ce5bee5d3eeb666d9e5f0514da01f9e913aa20e776ac618687de544400f4a835"} Jul 06 19:11:19.163208 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.163183 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-ktkpf" Jul 06 19:11:19.164686 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.164652 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" event={"ID":"e7cffb20-d0f2-4323-bc35-e5695a8339f0","Type":"ContainerStarted","Data":"6028537d33681244d9298d0392deb8f968b8abc656da9dc68ae07bd84be594d1"} Jul 06 19:11:19.165046 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.165018 2554 patch_prober.go:28] interesting pod/downloads-685bfb5f64-ktkpf container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" start-of-body= Jul 06 19:11:19.165172 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.165073 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-ktkpf" podUID="fabc8a56-c8b0-4def-9e61-5b5430ac4538" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.15:8080/\": dial tcp 10.133.0.15:8080: connect: connection refused" Jul 06 19:11:19.166961 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.166867 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b68f7cb55-hstjc" event={"ID":"42d85f11-b3e9-48bb-8dfd-eb9c9490b255","Type":"ContainerStarted","Data":"cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2"} Jul 06 19:11:19.166961 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.166913 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b68f7cb55-hstjc" event={"ID":"42d85f11-b3e9-48bb-8dfd-eb9c9490b255","Type":"ContainerStarted","Data":"fc1517e904af2935a496fc9157b8c7ad5c447c75f1a1d22a12068c4568b688cb"} Jul 06 19:11:19.169936 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.169839 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" event={"ID":"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81","Type":"ContainerStarted","Data":"bc5e3530ae034878e903e16b66eaa1aef96522c982a81e1d5ebb9bef070755c4"} Jul 06 19:11:19.172152 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.172100 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79aa8a2b-7643-4a49-b603-57fb45202849","Type":"ContainerStarted","Data":"c3954d6f7fd2506e6afabb0cf4887371e43d09408153d7d7e9f9778cc76a8a58"} Jul 06 19:11:19.183848 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.183612 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-ktkpf" podStartSLOduration=1.52965309 podStartE2EDuration="28.183592491s" totalImagesPullingTime="26.653939401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:19.181023236 +0000 UTC m=+104.664246830" watchObservedRunningTime="2026-07-06 19:11:19.183592491 +0000 UTC m=+104.666816077" Jul 06 19:11:19.200466 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:19.200412 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b68f7cb55-hstjc" podStartSLOduration=7.200394877 podStartE2EDuration="7.200394877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:19.200219178 +0000 UTC m=+104.683442753" watchObservedRunningTime="2026-07-06 19:11:19.200394877 +0000 UTC m=+104.683618449" Jul 06 19:11:20.202422 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:20.202388 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-ktkpf" Jul 06 19:11:22.189473 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:22.189382 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" event={"ID":"b6e1ac1c-7d0a-4306-ad97-2639ab10ed81","Type":"ContainerStarted","Data":"01b97b516bc85556bb62490b3939e520f9633c1a65e32c193cd832f0bea99b81"} Jul 06 19:11:22.191835 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:22.191806 2554 generic.go:356] "Generic (PLEG): container finished" podID="79aa8a2b-7643-4a49-b603-57fb45202849" containerID="44838809c55acf91b185811bff10363898a6f4fe34eee4a1140ff919ce142309" exitCode=0 Jul 06 19:11:22.191939 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:22.191858 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79aa8a2b-7643-4a49-b603-57fb45202849","Type":"ContainerDied","Data":"44838809c55acf91b185811bff10363898a6f4fe34eee4a1140ff919ce142309"} Jul 06 19:11:22.194503 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:22.194481 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" event={"ID":"e7cffb20-d0f2-4323-bc35-e5695a8339f0","Type":"ContainerStarted","Data":"1bc7b3515af43b95f2a8cddcb1cb8b7e21d5e18b618a2d270ec048357ec7f7d8"} Jul 06 19:11:22.209872 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:22.209804 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" podStartSLOduration=7.758665139 podStartE2EDuration="11.209789114s" totalImagesPullingTime="3.451123975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:22.208044921 +0000 UTC m=+107.691268513" watchObservedRunningTime="2026-07-06 19:11:22.209789114 +0000 UTC m=+107.693012687" Jul 06 19:11:22.223588 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:22.223540 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" podStartSLOduration=7.939209747 podStartE2EDuration="11.223524061s" totalImagesPullingTime="3.284314314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:22.223176989 +0000 UTC m=+107.706400562" watchObservedRunningTime="2026-07-06 19:11:22.223524061 +0000 UTC m=+107.706747634" Jul 06 19:11:22.603714 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:22.603673 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:22.603919 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:22.603767 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:22.609266 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:22.609241 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:23.026096 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:23.025953 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g5gwx" Jul 06 19:11:23.201434 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:23.201360 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:23.207888 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:23.207657 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:11:23.207888 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:23.207826 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-6bgpn" Jul 06 19:11:23.265351 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:23.264570 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c9f48d9d8-ccxpt"] Jul 06 19:11:27.224219 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:27.224125 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79aa8a2b-7643-4a49-b603-57fb45202849","Type":"ContainerStarted","Data":"75c1177bc718377c34add76497e3b6ee3b8cfe16b23ac8242500169352420703"} Jul 06 19:11:27.224219 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:27.224203 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79aa8a2b-7643-4a49-b603-57fb45202849","Type":"ContainerStarted","Data":"f9ce8b63a9bd9458d1ffc1e575afefe51a5d7889e652abb0c30e703a3040f653"} Jul 06 19:11:30.167915 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.167854 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6b74c8cf7b-6f6x5" podUID="051be0d1-2510-4728-b6f6-c3d7381d16da" containerName="console" containerID="cri-o://e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d" gracePeriod=15 Jul 06 19:11:30.240248 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.240205 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79aa8a2b-7643-4a49-b603-57fb45202849","Type":"ContainerStarted","Data":"3a3409e60d8965a9bc36086b97f6371ca080854e2101ae1d72711534029bb608"} Jul 06 19:11:30.240404 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.240279 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79aa8a2b-7643-4a49-b603-57fb45202849","Type":"ContainerStarted","Data":"792171b10c073bcb6c43ae1c2f63ab1af5cd9713d4230efd82c950ca72778581"} Jul 06 19:11:30.240404 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.240304 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79aa8a2b-7643-4a49-b603-57fb45202849","Type":"ContainerStarted","Data":"815e781b95d57270003554f122ffe64224e15cd069ac857c9943ce45953bc415"} Jul 06 19:11:30.240404 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.240329 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"79aa8a2b-7643-4a49-b603-57fb45202849","Type":"ContainerStarted","Data":"4d06320d45eddaa6b21f17104daeaa6c9cdd353f0e7bc23e5c8d3d7c22107a16"} Jul 06 19:11:30.271124 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.271062 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=8.285207097 podStartE2EDuration="18.271040657s" totalImagesPullingTime="9.965849681s" totalInitContainerRuntime="19.983879ms" isStatefulPod=false podCreationTimestamp="2026-07-06 19:11:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:11:30.268165602 +0000 UTC m=+115.751389193" watchObservedRunningTime="2026-07-06 19:11:30.271040657 +0000 UTC m=+115.754264231" Jul 06 19:11:30.455532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.455450 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:11:30.529745 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.529707 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-console-config\") pod \"051be0d1-2510-4728-b6f6-c3d7381d16da\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " Jul 06 19:11:30.529917 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.529770 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-oauth-config\") pod \"051be0d1-2510-4728-b6f6-c3d7381d16da\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " Jul 06 19:11:30.529917 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.529814 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-serving-cert\") pod \"051be0d1-2510-4728-b6f6-c3d7381d16da\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " Jul 06 19:11:30.529917 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.529849 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-service-ca\") pod \"051be0d1-2510-4728-b6f6-c3d7381d16da\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " Jul 06 19:11:30.530079 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.529946 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-oauth-serving-cert\") pod \"051be0d1-2510-4728-b6f6-c3d7381d16da\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " Jul 06 19:11:30.530079 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.529985 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vrh2\" (UniqueName: \"kubernetes.io/projected/051be0d1-2510-4728-b6f6-c3d7381d16da-kube-api-access-8vrh2\") pod \"051be0d1-2510-4728-b6f6-c3d7381d16da\" (UID: \"051be0d1-2510-4728-b6f6-c3d7381d16da\") " Jul 06 19:11:30.530400 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.530208 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-console-config" (OuterVolumeSpecName: "console-config") pod "051be0d1-2510-4728-b6f6-c3d7381d16da" (UID: "051be0d1-2510-4728-b6f6-c3d7381d16da"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:11:30.530400 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.530350 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "051be0d1-2510-4728-b6f6-c3d7381d16da" (UID: "051be0d1-2510-4728-b6f6-c3d7381d16da"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:11:30.530400 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.530364 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-service-ca" (OuterVolumeSpecName: "service-ca") pod "051be0d1-2510-4728-b6f6-c3d7381d16da" (UID: "051be0d1-2510-4728-b6f6-c3d7381d16da"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:11:30.532236 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.532213 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "051be0d1-2510-4728-b6f6-c3d7381d16da" (UID: "051be0d1-2510-4728-b6f6-c3d7381d16da"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:11:30.532361 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.532235 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "051be0d1-2510-4728-b6f6-c3d7381d16da" (UID: "051be0d1-2510-4728-b6f6-c3d7381d16da"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:11:30.532361 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.532237 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051be0d1-2510-4728-b6f6-c3d7381d16da-kube-api-access-8vrh2" (OuterVolumeSpecName: "kube-api-access-8vrh2") pod "051be0d1-2510-4728-b6f6-c3d7381d16da" (UID: "051be0d1-2510-4728-b6f6-c3d7381d16da"). InnerVolumeSpecName "kube-api-access-8vrh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:11:30.630908 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.630871 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-oauth-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:30.630908 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.630902 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vrh2\" (UniqueName: \"kubernetes.io/projected/051be0d1-2510-4728-b6f6-c3d7381d16da-kube-api-access-8vrh2\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:30.630908 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.630912 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-console-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:30.631141 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.630920 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-oauth-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:30.631141 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.630929 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/051be0d1-2510-4728-b6f6-c3d7381d16da-console-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:30.631141 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:30.630937 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/051be0d1-2510-4728-b6f6-c3d7381d16da-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:31.245392 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.245353 2554 generic.go:356] "Generic (PLEG): container finished" podID="051be0d1-2510-4728-b6f6-c3d7381d16da" containerID="e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d" exitCode=0 Jul 06 19:11:31.245951 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.245399 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b74c8cf7b-6f6x5" Jul 06 19:11:31.245951 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.245434 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b74c8cf7b-6f6x5" event={"ID":"051be0d1-2510-4728-b6f6-c3d7381d16da","Type":"ContainerDied","Data":"e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d"} Jul 06 19:11:31.245951 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.245477 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b74c8cf7b-6f6x5" event={"ID":"051be0d1-2510-4728-b6f6-c3d7381d16da","Type":"ContainerDied","Data":"f323462f1d7cfd9f377a54ea75db5de7b3a9f659ca367c9e94e849d2788812e9"} Jul 06 19:11:31.245951 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.245499 2554 scope.go:117] "RemoveContainer" containerID="e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d" Jul 06 19:11:31.257693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.257677 2554 scope.go:117] "RemoveContainer" containerID="e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d" Jul 06 19:11:31.257966 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:11:31.257948 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d\": container with ID starting with e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d not found: ID does not exist" containerID="e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d" Jul 06 19:11:31.258020 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.257975 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d"} err="failed to get container status \"e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d\": rpc error: code = NotFound desc = could not find container \"e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d\": container with ID starting with e35ed5dca3565d50e78db0ecd16a90a672780ca341c47b4290d6f570154f140d not found: ID does not exist" Jul 06 19:11:31.266397 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.266372 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b74c8cf7b-6f6x5"] Jul 06 19:11:31.270023 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.270006 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b74c8cf7b-6f6x5"] Jul 06 19:11:31.454340 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.454299 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:31.454515 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:31.454364 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:32.529483 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:32.529450 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="051be0d1-2510-4728-b6f6-c3d7381d16da" path="/var/lib/kubelet/pods/051be0d1-2510-4728-b6f6-c3d7381d16da/volumes" Jul 06 19:11:33.212143 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:33.212115 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:11:33.256157 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:33.256119 2554 generic.go:356] "Generic (PLEG): container finished" podID="cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a" containerID="d9ac0fcee06c77df737fbd16e79fcd547f2150945de43a8db7fbbcbf00eda781" exitCode=0 Jul 06 19:11:33.256291 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:33.256194 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" event={"ID":"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a","Type":"ContainerDied","Data":"d9ac0fcee06c77df737fbd16e79fcd547f2150945de43a8db7fbbcbf00eda781"} Jul 06 19:11:33.256530 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:33.256511 2554 scope.go:117] "RemoveContainer" containerID="d9ac0fcee06c77df737fbd16e79fcd547f2150945de43a8db7fbbcbf00eda781" Jul 06 19:11:34.263966 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:34.263928 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mdnbf" event={"ID":"cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a","Type":"ContainerStarted","Data":"cb22e2966ef259a8df48bbbd2bc6510c4b7602979a5788b41c8bef826eb8f9ed"} Jul 06 19:11:38.671849 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:38.671784 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" podUID="cb102f4e-c53d-459c-aaed-9b20db1d13a6" containerName="registry" containerID="cri-o://a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14" gracePeriod=30 Jul 06 19:11:38.968592 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:38.968564 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:11:39.110398 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.110362 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-certificates\") pod \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " Jul 06 19:11:39.110569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.110430 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-installation-pull-secrets\") pod \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " Jul 06 19:11:39.110569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.110486 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-image-registry-private-configuration\") pod \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " Jul 06 19:11:39.110569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.110522 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") pod \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " Jul 06 19:11:39.110569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.110546 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb102f4e-c53d-459c-aaed-9b20db1d13a6-ca-trust-extracted\") pod \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " Jul 06 19:11:39.110780 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.110604 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-trusted-ca\") pod \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " Jul 06 19:11:39.110780 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.110659 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c7sb\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-kube-api-access-5c7sb\") pod \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " Jul 06 19:11:39.110780 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.110693 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-bound-sa-token\") pod \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\" (UID: \"cb102f4e-c53d-459c-aaed-9b20db1d13a6\") " Jul 06 19:11:39.110936 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.110834 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cb102f4e-c53d-459c-aaed-9b20db1d13a6" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:11:39.111207 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.111162 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cb102f4e-c53d-459c-aaed-9b20db1d13a6" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:11:39.111207 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.111167 2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-certificates\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:39.120042 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.120008 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cb102f4e-c53d-459c-aaed-9b20db1d13a6" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:11:39.120176 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.120034 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cb102f4e-c53d-459c-aaed-9b20db1d13a6" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:11:39.120176 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.120068 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cb102f4e-c53d-459c-aaed-9b20db1d13a6" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:11:39.120176 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.120127 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-kube-api-access-5c7sb" (OuterVolumeSpecName: "kube-api-access-5c7sb") pod "cb102f4e-c53d-459c-aaed-9b20db1d13a6" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6"). InnerVolumeSpecName "kube-api-access-5c7sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:11:39.120315 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.120221 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cb102f4e-c53d-459c-aaed-9b20db1d13a6" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:11:39.122066 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.122041 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb102f4e-c53d-459c-aaed-9b20db1d13a6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cb102f4e-c53d-459c-aaed-9b20db1d13a6" (UID: "cb102f4e-c53d-459c-aaed-9b20db1d13a6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:11:39.211998 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.211923 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb102f4e-c53d-459c-aaed-9b20db1d13a6-trusted-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:39.211998 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.211951 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c7sb\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-kube-api-access-5c7sb\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:39.211998 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.211961 2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-bound-sa-token\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:39.211998 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.211969 2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-installation-pull-secrets\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:39.211998 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.211979 2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb102f4e-c53d-459c-aaed-9b20db1d13a6-image-registry-private-configuration\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:39.211998 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.211988 2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb102f4e-c53d-459c-aaed-9b20db1d13a6-registry-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:39.211998 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.211996 2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb102f4e-c53d-459c-aaed-9b20db1d13a6-ca-trust-extracted\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:39.284521 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.284485 2554 generic.go:356] "Generic (PLEG): container finished" podID="cb102f4e-c53d-459c-aaed-9b20db1d13a6" containerID="a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14" exitCode=0 Jul 06 19:11:39.284678 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.284541 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" Jul 06 19:11:39.284678 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.284570 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" event={"ID":"cb102f4e-c53d-459c-aaed-9b20db1d13a6","Type":"ContainerDied","Data":"a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14"} Jul 06 19:11:39.284678 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.284646 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84995c7cbf-v2kvf" event={"ID":"cb102f4e-c53d-459c-aaed-9b20db1d13a6","Type":"ContainerDied","Data":"43618059c8f4c665e243d872f568e73f7795927b90ff5fc5cbe97260dda67785"} Jul 06 19:11:39.284678 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.284669 2554 scope.go:117] "RemoveContainer" containerID="a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14" Jul 06 19:11:39.293835 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.293808 2554 scope.go:117] "RemoveContainer" containerID="a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14" Jul 06 19:11:39.294077 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:11:39.294057 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14\": container with ID starting with a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14 not found: ID does not exist" containerID="a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14" Jul 06 19:11:39.294144 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.294084 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14"} err="failed to get container status \"a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14\": rpc error: code = NotFound desc = could not find container \"a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14\": container with ID starting with a89a76629ed2aedddcf302be26c6f4b8b62a34c0970d96463ead575ec3ae8c14 not found: ID does not exist" Jul 06 19:11:39.305622 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.305601 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84995c7cbf-v2kvf"] Jul 06 19:11:39.308605 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:39.308586 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84995c7cbf-v2kvf"] Jul 06 19:11:40.529824 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:40.529791 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb102f4e-c53d-459c-aaed-9b20db1d13a6" path="/var/lib/kubelet/pods/cb102f4e-c53d-459c-aaed-9b20db1d13a6/volumes" Jul 06 19:11:48.304295 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.304219 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6c9f48d9d8-ccxpt" podUID="dcf309f5-cd5f-44df-848b-9a3c21381aa8" containerName="console" containerID="cri-o://dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce" gracePeriod=15 Jul 06 19:11:48.577193 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.577168 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:11:48.587875 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.587852 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-oauth-config\") pod \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " Jul 06 19:11:48.588035 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.587885 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-serving-cert\") pod \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " Jul 06 19:11:48.588035 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.587917 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-oauth-serving-cert\") pod \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " Jul 06 19:11:48.588035 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.587967 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-service-ca\") pod \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " Jul 06 19:11:48.588035 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.587982 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-config\") pod \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " Jul 06 19:11:48.588035 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.588031 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-trusted-ca-bundle\") pod \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " Jul 06 19:11:48.588417 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.588387 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-config" (OuterVolumeSpecName: "console-config") pod "dcf309f5-cd5f-44df-848b-9a3c21381aa8" (UID: "dcf309f5-cd5f-44df-848b-9a3c21381aa8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:11:48.588532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.588409 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-service-ca" (OuterVolumeSpecName: "service-ca") pod "dcf309f5-cd5f-44df-848b-9a3c21381aa8" (UID: "dcf309f5-cd5f-44df-848b-9a3c21381aa8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:11:48.588532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.588396 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dcf309f5-cd5f-44df-848b-9a3c21381aa8" (UID: "dcf309f5-cd5f-44df-848b-9a3c21381aa8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:11:48.588532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.588501 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dcf309f5-cd5f-44df-848b-9a3c21381aa8" (UID: "dcf309f5-cd5f-44df-848b-9a3c21381aa8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:11:48.590141 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.590113 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dcf309f5-cd5f-44df-848b-9a3c21381aa8" (UID: "dcf309f5-cd5f-44df-848b-9a3c21381aa8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:11:48.590271 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.590254 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dcf309f5-cd5f-44df-848b-9a3c21381aa8" (UID: "dcf309f5-cd5f-44df-848b-9a3c21381aa8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:11:48.688926 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.688890 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mdf6\" (UniqueName: \"kubernetes.io/projected/dcf309f5-cd5f-44df-848b-9a3c21381aa8-kube-api-access-7mdf6\") pod \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\" (UID: \"dcf309f5-cd5f-44df-848b-9a3c21381aa8\") " Jul 06 19:11:48.689111 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.689050 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:48.689111 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.689060 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-trusted-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:48.689111 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.689069 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-oauth-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:48.689111 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.689077 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dcf309f5-cd5f-44df-848b-9a3c21381aa8-console-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:48.689111 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.689085 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-oauth-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:48.689111 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.689092 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf309f5-cd5f-44df-848b-9a3c21381aa8-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:48.690909 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.690882 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf309f5-cd5f-44df-848b-9a3c21381aa8-kube-api-access-7mdf6" (OuterVolumeSpecName: "kube-api-access-7mdf6") pod "dcf309f5-cd5f-44df-848b-9a3c21381aa8" (UID: "dcf309f5-cd5f-44df-848b-9a3c21381aa8"). InnerVolumeSpecName "kube-api-access-7mdf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:11:48.790044 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:48.790005 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mdf6\" (UniqueName: \"kubernetes.io/projected/dcf309f5-cd5f-44df-848b-9a3c21381aa8-kube-api-access-7mdf6\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:11:49.325478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:49.325437 2554 generic.go:356] "Generic (PLEG): container finished" podID="dcf309f5-cd5f-44df-848b-9a3c21381aa8" containerID="dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce" exitCode=0 Jul 06 19:11:49.325478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:49.325484 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9f48d9d8-ccxpt" Jul 06 19:11:49.326057 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:49.325527 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9f48d9d8-ccxpt" event={"ID":"dcf309f5-cd5f-44df-848b-9a3c21381aa8","Type":"ContainerDied","Data":"dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce"} Jul 06 19:11:49.326057 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:49.325591 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9f48d9d8-ccxpt" event={"ID":"dcf309f5-cd5f-44df-848b-9a3c21381aa8","Type":"ContainerDied","Data":"79b653b3bd7987b861b43cc45b756180f136410150a3776aced77b7742c9e5ab"} Jul 06 19:11:49.326057 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:49.325619 2554 scope.go:117] "RemoveContainer" containerID="dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce" Jul 06 19:11:49.334541 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:49.334525 2554 scope.go:117] "RemoveContainer" containerID="dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce" Jul 06 19:11:49.334819 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:11:49.334801 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce\": container with ID starting with dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce not found: ID does not exist" containerID="dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce" Jul 06 19:11:49.334867 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:49.334829 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce"} err="failed to get container status \"dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce\": rpc error: code = NotFound desc = could not find container \"dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce\": container with ID starting with dc6fe93314de8360cd9300db43c851ad9a1f747bda1218335a435df1df8ad3ce not found: ID does not exist" Jul 06 19:11:49.351138 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:49.351113 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c9f48d9d8-ccxpt"] Jul 06 19:11:49.357511 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:49.357489 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c9f48d9d8-ccxpt"] Jul 06 19:11:50.529554 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:50.529516 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf309f5-cd5f-44df-848b-9a3c21381aa8" path="/var/lib/kubelet/pods/dcf309f5-cd5f-44df-848b-9a3c21381aa8/volumes" Jul 06 19:11:51.459699 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:51.459672 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:51.463337 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:51.463316 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-55495fc5d5-fpvk6" Jul 06 19:11:53.343919 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:53.343886 2554 generic.go:356] "Generic (PLEG): container finished" podID="11ca926e-8858-4b62-b157-62833f8411e1" containerID="bc77abe18b11732eb956b9a94564b88e0bde4ef547de7e16d718ff651df88473" exitCode=0 Jul 06 19:11:53.344323 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:53.343963 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-58r22" event={"ID":"11ca926e-8858-4b62-b157-62833f8411e1","Type":"ContainerDied","Data":"bc77abe18b11732eb956b9a94564b88e0bde4ef547de7e16d718ff651df88473"} Jul 06 19:11:53.344323 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:53.344307 2554 scope.go:117] "RemoveContainer" containerID="bc77abe18b11732eb956b9a94564b88e0bde4ef547de7e16d718ff651df88473" Jul 06 19:11:54.349928 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:11:54.349887 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-58r22" event={"ID":"11ca926e-8858-4b62-b157-62833f8411e1","Type":"ContainerStarted","Data":"182e1a567b79e7df7ebd8525b7666c61ee5eefdcdd1094bf1ab1406a1baeb1ad"} Jul 06 19:12:13.212198 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:13.212161 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:12:13.275052 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:13.275022 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:12:13.443351 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:13.443325 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 19:12:34.901885 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.901804 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57d456d769-8xvnm"] Jul 06 19:12:34.902332 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.902213 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb102f4e-c53d-459c-aaed-9b20db1d13a6" containerName="registry" Jul 06 19:12:34.902332 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.902228 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb102f4e-c53d-459c-aaed-9b20db1d13a6" containerName="registry" Jul 06 19:12:34.902332 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.902240 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="051be0d1-2510-4728-b6f6-c3d7381d16da" containerName="console" Jul 06 19:12:34.902332 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.902246 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="051be0d1-2510-4728-b6f6-c3d7381d16da" containerName="console" Jul 06 19:12:34.902332 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.902258 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf309f5-cd5f-44df-848b-9a3c21381aa8" containerName="console" Jul 06 19:12:34.902332 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.902264 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf309f5-cd5f-44df-848b-9a3c21381aa8" containerName="console" Jul 06 19:12:34.902332 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.902319 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="051be0d1-2510-4728-b6f6-c3d7381d16da" containerName="console" Jul 06 19:12:34.902332 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.902328 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf309f5-cd5f-44df-848b-9a3c21381aa8" containerName="console" Jul 06 19:12:34.902332 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.902334 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb102f4e-c53d-459c-aaed-9b20db1d13a6" containerName="registry" Jul 06 19:12:34.904828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.904810 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:34.917866 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.917839 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d456d769-8xvnm"] Jul 06 19:12:34.983006 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.982965 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-service-ca\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:34.983006 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.983002 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-trusted-ca-bundle\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:34.983270 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.983031 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-console-config\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:34.983270 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.983098 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-oauth-serving-cert\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:34.983270 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.983156 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-oauth-config\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:34.983270 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.983202 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9sd\" (UniqueName: \"kubernetes.io/projected/95711e88-5bd1-4e94-a4c5-2febb234d059-kube-api-access-7n9sd\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:34.983411 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:34.983278 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-serving-cert\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.084287 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.084237 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-console-config\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.084287 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.084291 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-oauth-serving-cert\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.084560 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.084313 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-oauth-config\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.084560 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.084354 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9sd\" (UniqueName: \"kubernetes.io/projected/95711e88-5bd1-4e94-a4c5-2febb234d059-kube-api-access-7n9sd\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.084560 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.084417 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-serving-cert\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.084560 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.084470 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-service-ca\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.084560 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.084491 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-trusted-ca-bundle\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.085268 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.085226 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-oauth-serving-cert\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.085395 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.085362 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-service-ca\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.085549 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.085530 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-console-config\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.085748 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.085716 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-trusted-ca-bundle\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.087037 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.087016 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-serving-cert\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.087205 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.087185 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-oauth-config\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.092873 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.092844 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9sd\" (UniqueName: \"kubernetes.io/projected/95711e88-5bd1-4e94-a4c5-2febb234d059-kube-api-access-7n9sd\") pod \"console-57d456d769-8xvnm\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.213811 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.213692 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:35.342589 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.342550 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57d456d769-8xvnm"] Jul 06 19:12:35.345623 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:12:35.345588 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95711e88_5bd1_4e94_a4c5_2febb234d059.slice/crio-f1ad0262819cdb062fb4ac035a959977863d86208cece0176328231c687ee46e WatchSource:0}: Error finding container f1ad0262819cdb062fb4ac035a959977863d86208cece0176328231c687ee46e: Status 404 returned error can't find the container with id f1ad0262819cdb062fb4ac035a959977863d86208cece0176328231c687ee46e Jul 06 19:12:35.519274 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.519178 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d456d769-8xvnm" event={"ID":"95711e88-5bd1-4e94-a4c5-2febb234d059","Type":"ContainerStarted","Data":"803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7"} Jul 06 19:12:35.519274 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.519264 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d456d769-8xvnm" event={"ID":"95711e88-5bd1-4e94-a4c5-2febb234d059","Type":"ContainerStarted","Data":"f1ad0262819cdb062fb4ac035a959977863d86208cece0176328231c687ee46e"} Jul 06 19:12:35.535398 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:35.535332 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57d456d769-8xvnm" podStartSLOduration=1.535314802 podStartE2EDuration="1.535314802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:35.534785142 +0000 UTC m=+181.018008715" watchObservedRunningTime="2026-07-06 19:12:35.535314802 +0000 UTC m=+181.018538380" Jul 06 19:12:45.214697 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.214652 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:45.215215 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.214781 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:45.219545 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.219521 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:45.398420 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.398387 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d456d769-8xvnm"] Jul 06 19:12:45.427873 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.427824 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c4c6fb4cc-t89wk"] Jul 06 19:12:45.431196 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.431175 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.441787 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.441762 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4c6fb4cc-t89wk"] Jul 06 19:12:45.479803 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.479684 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-serving-cert\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.479803 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.479720 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-trusted-ca-bundle\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.479803 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.479785 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-service-ca\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.480036 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.479820 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q76ng\" (UniqueName: \"kubernetes.io/projected/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-kube-api-access-q76ng\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.480036 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.479862 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-oauth-serving-cert\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.480036 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.479891 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-oauth-config\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.480036 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.479919 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-config\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.561198 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.561168 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:12:45.580945 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.580907 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-oauth-serving-cert\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.580945 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.580948 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-oauth-config\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.581215 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.580962 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-config\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.581215 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.581121 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-serving-cert\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.581215 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.581148 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-trusted-ca-bundle\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.581215 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.581183 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-service-ca\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.581429 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.581219 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76ng\" (UniqueName: \"kubernetes.io/projected/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-kube-api-access-q76ng\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.581760 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.581710 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-oauth-serving-cert\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.581928 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.581764 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-config\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.582126 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.582103 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-service-ca\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.582792 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.582760 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-trusted-ca-bundle\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.583671 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.583650 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-serving-cert\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.583786 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.583768 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-oauth-config\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.589254 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.589236 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q76ng\" (UniqueName: \"kubernetes.io/projected/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-kube-api-access-q76ng\") pod \"console-6c4c6fb4cc-t89wk\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.740134 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.740023 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:45.873614 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:45.873578 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4c6fb4cc-t89wk"] Jul 06 19:12:45.877032 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:12:45.877001 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72e8db6_5f2d_4bd3_9007_e92c18ff0094.slice/crio-c71f21c72ab9b8d69a10d8b80aa89b041c812c654684dfed4c4616711af5ed49 WatchSource:0}: Error finding container c71f21c72ab9b8d69a10d8b80aa89b041c812c654684dfed4c4616711af5ed49: Status 404 returned error can't find the container with id c71f21c72ab9b8d69a10d8b80aa89b041c812c654684dfed4c4616711af5ed49 Jul 06 19:12:46.563107 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:46.563064 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4c6fb4cc-t89wk" event={"ID":"d72e8db6-5f2d-4bd3-9007-e92c18ff0094","Type":"ContainerStarted","Data":"d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40"} Jul 06 19:12:46.563537 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:46.563126 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4c6fb4cc-t89wk" event={"ID":"d72e8db6-5f2d-4bd3-9007-e92c18ff0094","Type":"ContainerStarted","Data":"c71f21c72ab9b8d69a10d8b80aa89b041c812c654684dfed4c4616711af5ed49"} Jul 06 19:12:46.582436 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:46.582382 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c4c6fb4cc-t89wk" podStartSLOduration=1.582367158 podStartE2EDuration="1.582367158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:46.580004186 +0000 UTC m=+192.063227758" watchObservedRunningTime="2026-07-06 19:12:46.582367158 +0000 UTC m=+192.065590730" Jul 06 19:12:55.741226 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:55.741182 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:55.741622 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:55.741255 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:55.745913 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:55.745893 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:56.609308 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:56.609279 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:12:56.654764 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:12:56.654721 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b68f7cb55-hstjc"] Jul 06 19:13:11.582703 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.582633 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-57d456d769-8xvnm" podUID="95711e88-5bd1-4e94-a4c5-2febb234d059" containerName="console" containerID="cri-o://803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7" gracePeriod=15 Jul 06 19:13:11.826308 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.826284 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:13:11.914258 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.914163 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-console-config\") pod \"95711e88-5bd1-4e94-a4c5-2febb234d059\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " Jul 06 19:13:11.914258 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.914202 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-oauth-serving-cert\") pod \"95711e88-5bd1-4e94-a4c5-2febb234d059\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " Jul 06 19:13:11.914258 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.914239 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-service-ca\") pod \"95711e88-5bd1-4e94-a4c5-2febb234d059\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " Jul 06 19:13:11.914532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.914282 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n9sd\" (UniqueName: \"kubernetes.io/projected/95711e88-5bd1-4e94-a4c5-2febb234d059-kube-api-access-7n9sd\") pod \"95711e88-5bd1-4e94-a4c5-2febb234d059\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " Jul 06 19:13:11.914532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.914326 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-trusted-ca-bundle\") pod \"95711e88-5bd1-4e94-a4c5-2febb234d059\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " Jul 06 19:13:11.914532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.914372 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-serving-cert\") pod \"95711e88-5bd1-4e94-a4c5-2febb234d059\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " Jul 06 19:13:11.914532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.914406 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-oauth-config\") pod \"95711e88-5bd1-4e94-a4c5-2febb234d059\" (UID: \"95711e88-5bd1-4e94-a4c5-2febb234d059\") " Jul 06 19:13:11.914775 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.914639 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-console-config" (OuterVolumeSpecName: "console-config") pod "95711e88-5bd1-4e94-a4c5-2febb234d059" (UID: "95711e88-5bd1-4e94-a4c5-2febb234d059"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:13:11.914775 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.914666 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-service-ca" (OuterVolumeSpecName: "service-ca") pod "95711e88-5bd1-4e94-a4c5-2febb234d059" (UID: "95711e88-5bd1-4e94-a4c5-2febb234d059"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:13:11.914775 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.914680 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "95711e88-5bd1-4e94-a4c5-2febb234d059" (UID: "95711e88-5bd1-4e94-a4c5-2febb234d059"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:13:11.915035 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.915006 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "95711e88-5bd1-4e94-a4c5-2febb234d059" (UID: "95711e88-5bd1-4e94-a4c5-2febb234d059"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:13:11.916593 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.916560 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "95711e88-5bd1-4e94-a4c5-2febb234d059" (UID: "95711e88-5bd1-4e94-a4c5-2febb234d059"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:13:11.916698 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.916596 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "95711e88-5bd1-4e94-a4c5-2febb234d059" (UID: "95711e88-5bd1-4e94-a4c5-2febb234d059"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:13:11.916698 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:11.916597 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95711e88-5bd1-4e94-a4c5-2febb234d059-kube-api-access-7n9sd" (OuterVolumeSpecName: "kube-api-access-7n9sd") pod "95711e88-5bd1-4e94-a4c5-2febb234d059" (UID: "95711e88-5bd1-4e94-a4c5-2febb234d059"). InnerVolumeSpecName "kube-api-access-7n9sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:13:12.015854 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.015819 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:12.015854 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.015848 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/95711e88-5bd1-4e94-a4c5-2febb234d059-console-oauth-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:12.015854 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.015858 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-console-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:12.016087 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.015868 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-oauth-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:12.016087 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.015877 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:12.016087 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.015885 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n9sd\" (UniqueName: \"kubernetes.io/projected/95711e88-5bd1-4e94-a4c5-2febb234d059-kube-api-access-7n9sd\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:12.016087 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.015901 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95711e88-5bd1-4e94-a4c5-2febb234d059-trusted-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:12.669420 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.669380 2554 generic.go:356] "Generic (PLEG): container finished" podID="95711e88-5bd1-4e94-a4c5-2febb234d059" containerID="803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7" exitCode=0 Jul 06 19:13:12.669927 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.669428 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57d456d769-8xvnm" Jul 06 19:13:12.669927 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.669425 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d456d769-8xvnm" event={"ID":"95711e88-5bd1-4e94-a4c5-2febb234d059","Type":"ContainerDied","Data":"803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7"} Jul 06 19:13:12.669927 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.669492 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57d456d769-8xvnm" event={"ID":"95711e88-5bd1-4e94-a4c5-2febb234d059","Type":"ContainerDied","Data":"f1ad0262819cdb062fb4ac035a959977863d86208cece0176328231c687ee46e"} Jul 06 19:13:12.669927 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.669520 2554 scope.go:117] "RemoveContainer" containerID="803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7" Jul 06 19:13:12.678070 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.678056 2554 scope.go:117] "RemoveContainer" containerID="803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7" Jul 06 19:13:12.678306 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:13:12.678290 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7\": container with ID starting with 803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7 not found: ID does not exist" containerID="803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7" Jul 06 19:13:12.678352 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.678313 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7"} err="failed to get container status \"803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7\": rpc error: code = NotFound desc = could not find container \"803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7\": container with ID starting with 803c25d6bf960c425c72c6da071c8af7cc47d46b3327772b934428d47c3fd6f7 not found: ID does not exist" Jul 06 19:13:12.686301 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.686281 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57d456d769-8xvnm"] Jul 06 19:13:12.690043 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:12.690024 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57d456d769-8xvnm"] Jul 06 19:13:14.528628 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:14.528551 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95711e88-5bd1-4e94-a4c5-2febb234d059" path="/var/lib/kubelet/pods/95711e88-5bd1-4e94-a4c5-2febb234d059/volumes" Jul 06 19:13:21.683652 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:21.683595 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7b68f7cb55-hstjc" podUID="42d85f11-b3e9-48bb-8dfd-eb9c9490b255" containerName="console" containerID="cri-o://cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2" gracePeriod=15 Jul 06 19:13:21.922349 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:21.922323 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:13:22.000357 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.000257 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-oauth-config\") pod \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " Jul 06 19:13:22.000357 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.000311 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-serving-cert\") pod \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " Jul 06 19:13:22.000357 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.000350 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dlwf\" (UniqueName: \"kubernetes.io/projected/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-kube-api-access-5dlwf\") pod \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " Jul 06 19:13:22.000643 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.000392 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-trusted-ca-bundle\") pod \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " Jul 06 19:13:22.000643 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.000529 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-config\") pod \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " Jul 06 19:13:22.000643 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.000561 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-oauth-serving-cert\") pod \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " Jul 06 19:13:22.000643 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.000605 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-service-ca\") pod \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\" (UID: \"42d85f11-b3e9-48bb-8dfd-eb9c9490b255\") " Jul 06 19:13:22.000875 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.000854 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "42d85f11-b3e9-48bb-8dfd-eb9c9490b255" (UID: "42d85f11-b3e9-48bb-8dfd-eb9c9490b255"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:13:22.000931 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.000905 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-config" (OuterVolumeSpecName: "console-config") pod "42d85f11-b3e9-48bb-8dfd-eb9c9490b255" (UID: "42d85f11-b3e9-48bb-8dfd-eb9c9490b255"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:13:22.000991 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.000974 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-trusted-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:22.001181 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.001160 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "42d85f11-b3e9-48bb-8dfd-eb9c9490b255" (UID: "42d85f11-b3e9-48bb-8dfd-eb9c9490b255"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:13:22.001239 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.001200 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-service-ca" (OuterVolumeSpecName: "service-ca") pod "42d85f11-b3e9-48bb-8dfd-eb9c9490b255" (UID: "42d85f11-b3e9-48bb-8dfd-eb9c9490b255"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:13:22.002682 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.002654 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "42d85f11-b3e9-48bb-8dfd-eb9c9490b255" (UID: "42d85f11-b3e9-48bb-8dfd-eb9c9490b255"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:13:22.002935 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.002914 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "42d85f11-b3e9-48bb-8dfd-eb9c9490b255" (UID: "42d85f11-b3e9-48bb-8dfd-eb9c9490b255"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:13:22.002995 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.002978 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-kube-api-access-5dlwf" (OuterVolumeSpecName: "kube-api-access-5dlwf") pod "42d85f11-b3e9-48bb-8dfd-eb9c9490b255" (UID: "42d85f11-b3e9-48bb-8dfd-eb9c9490b255"). InnerVolumeSpecName "kube-api-access-5dlwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:13:22.101861 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.101818 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:22.101861 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.101851 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-oauth-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:22.101861 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.101860 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:22.101861 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.101869 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dlwf\" (UniqueName: \"kubernetes.io/projected/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-kube-api-access-5dlwf\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:22.102126 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.101878 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-console-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:22.102126 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.101886 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42d85f11-b3e9-48bb-8dfd-eb9c9490b255-oauth-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:13:22.712217 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.712182 2554 generic.go:356] "Generic (PLEG): container finished" podID="42d85f11-b3e9-48bb-8dfd-eb9c9490b255" containerID="cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2" exitCode=0 Jul 06 19:13:22.712603 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.712227 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b68f7cb55-hstjc" Jul 06 19:13:22.712603 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.712248 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b68f7cb55-hstjc" event={"ID":"42d85f11-b3e9-48bb-8dfd-eb9c9490b255","Type":"ContainerDied","Data":"cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2"} Jul 06 19:13:22.712603 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.712290 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b68f7cb55-hstjc" event={"ID":"42d85f11-b3e9-48bb-8dfd-eb9c9490b255","Type":"ContainerDied","Data":"fc1517e904af2935a496fc9157b8c7ad5c447c75f1a1d22a12068c4568b688cb"} Jul 06 19:13:22.712603 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.712308 2554 scope.go:117] "RemoveContainer" containerID="cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2" Jul 06 19:13:22.720487 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.720468 2554 scope.go:117] "RemoveContainer" containerID="cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2" Jul 06 19:13:22.720780 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:13:22.720760 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2\": container with ID starting with cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2 not found: ID does not exist" containerID="cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2" Jul 06 19:13:22.720844 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.720789 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2"} err="failed to get container status \"cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2\": rpc error: code = NotFound desc = could not find container \"cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2\": container with ID starting with cb4e33b470796dcd8aabf1ad4e25cb390d7fe769848fd4e7dde445d3e5012be2 not found: ID does not exist" Jul 06 19:13:22.728056 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.728027 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b68f7cb55-hstjc"] Jul 06 19:13:22.731498 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:22.731473 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b68f7cb55-hstjc"] Jul 06 19:13:24.528844 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:13:24.528812 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42d85f11-b3e9-48bb-8dfd-eb9c9490b255" path="/var/lib/kubelet/pods/42d85f11-b3e9-48bb-8dfd-eb9c9490b255/volumes" Jul 06 19:14:09.338747 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.338684 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f9ndc"] Jul 06 19:14:09.345699 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.345555 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95711e88-5bd1-4e94-a4c5-2febb234d059" containerName="console" Jul 06 19:14:09.345885 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.345705 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="95711e88-5bd1-4e94-a4c5-2febb234d059" containerName="console" Jul 06 19:14:09.345885 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.345765 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d85f11-b3e9-48bb-8dfd-eb9c9490b255" containerName="console" Jul 06 19:14:09.345885 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.345772 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d85f11-b3e9-48bb-8dfd-eb9c9490b255" containerName="console" Jul 06 19:14:09.345885 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.345834 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d85f11-b3e9-48bb-8dfd-eb9c9490b255" containerName="console" Jul 06 19:14:09.345885 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.345843 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="95711e88-5bd1-4e94-a4c5-2febb234d059" containerName="console" Jul 06 19:14:09.348449 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.348427 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f9ndc" Jul 06 19:14:09.351703 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.351678 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:14:09.351830 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.351796 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:14:09.353081 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.353060 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-jlhpb\"" Jul 06 19:14:09.355198 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.355176 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f9ndc"] Jul 06 19:14:09.480147 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.480111 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e02914d-7023-4732-8328-324aedde8085-bound-sa-token\") pod \"cert-manager-55b4d954d5-f9ndc\" (UID: \"8e02914d-7023-4732-8328-324aedde8085\") " pod="cert-manager/cert-manager-55b4d954d5-f9ndc" Jul 06 19:14:09.480325 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.480165 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lglj9\" (UniqueName: \"kubernetes.io/projected/8e02914d-7023-4732-8328-324aedde8085-kube-api-access-lglj9\") pod \"cert-manager-55b4d954d5-f9ndc\" (UID: \"8e02914d-7023-4732-8328-324aedde8085\") " pod="cert-manager/cert-manager-55b4d954d5-f9ndc" Jul 06 19:14:09.580802 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.580764 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e02914d-7023-4732-8328-324aedde8085-bound-sa-token\") pod \"cert-manager-55b4d954d5-f9ndc\" (UID: \"8e02914d-7023-4732-8328-324aedde8085\") " pod="cert-manager/cert-manager-55b4d954d5-f9ndc" Jul 06 19:14:09.580980 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.580824 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lglj9\" (UniqueName: \"kubernetes.io/projected/8e02914d-7023-4732-8328-324aedde8085-kube-api-access-lglj9\") pod \"cert-manager-55b4d954d5-f9ndc\" (UID: \"8e02914d-7023-4732-8328-324aedde8085\") " pod="cert-manager/cert-manager-55b4d954d5-f9ndc" Jul 06 19:14:09.589204 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.589128 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8e02914d-7023-4732-8328-324aedde8085-bound-sa-token\") pod \"cert-manager-55b4d954d5-f9ndc\" (UID: \"8e02914d-7023-4732-8328-324aedde8085\") " pod="cert-manager/cert-manager-55b4d954d5-f9ndc" Jul 06 19:14:09.589344 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.589271 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lglj9\" (UniqueName: \"kubernetes.io/projected/8e02914d-7023-4732-8328-324aedde8085-kube-api-access-lglj9\") pod \"cert-manager-55b4d954d5-f9ndc\" (UID: \"8e02914d-7023-4732-8328-324aedde8085\") " pod="cert-manager/cert-manager-55b4d954d5-f9ndc" Jul 06 19:14:09.670355 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.670305 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f9ndc" Jul 06 19:14:09.795896 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.795845 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f9ndc"] Jul 06 19:14:09.798699 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:14:09.798671 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e02914d_7023_4732_8328_324aedde8085.slice/crio-46a23dae5e2868a44a56757ea86da50c1f70efa3176f0815719d1bd2980c3dca WatchSource:0}: Error finding container 46a23dae5e2868a44a56757ea86da50c1f70efa3176f0815719d1bd2980c3dca: Status 404 returned error can't find the container with id 46a23dae5e2868a44a56757ea86da50c1f70efa3176f0815719d1bd2980c3dca Jul 06 19:14:09.897165 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:09.897076 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f9ndc" event={"ID":"8e02914d-7023-4732-8328-324aedde8085","Type":"ContainerStarted","Data":"46a23dae5e2868a44a56757ea86da50c1f70efa3176f0815719d1bd2980c3dca"} Jul 06 19:14:13.916814 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:13.916780 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f9ndc" event={"ID":"8e02914d-7023-4732-8328-324aedde8085","Type":"ContainerStarted","Data":"9d879d833b371be124d6d396be9e09af34dd65994b81a9792f5db7a84bdfea10"} Jul 06 19:14:13.933198 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:13.933154 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-f9ndc" podStartSLOduration=1.594823492 podStartE2EDuration="4.933137644s" totalImagesPullingTime="3.338314152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:14:13.931675163 +0000 UTC m=+279.414898737" watchObservedRunningTime="2026-07-06 19:14:13.933137644 +0000 UTC m=+279.416361216" Jul 06 19:14:34.582637 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.582598 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz"] Jul 06 19:14:34.584991 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.584974 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.588209 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.588185 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 19:14:34.588209 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.588199 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:14:34.589541 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.589523 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ckxcj\"" Jul 06 19:14:34.589597 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.589571 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 19:14:34.589642 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.589524 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:14:34.589677 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.589648 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 19:14:34.596266 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.596244 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz"] Jul 06 19:14:34.694579 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.694541 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6dfcb6ba-9dbd-4559-884e-a31a3989de13-metrics-cert\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.694803 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.694594 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6dfcb6ba-9dbd-4559-884e-a31a3989de13-manager-config\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.694803 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.694647 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lb96\" (UniqueName: \"kubernetes.io/projected/6dfcb6ba-9dbd-4559-884e-a31a3989de13-kube-api-access-4lb96\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.694803 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.694698 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dfcb6ba-9dbd-4559-884e-a31a3989de13-cert\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.795707 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.795676 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dfcb6ba-9dbd-4559-884e-a31a3989de13-cert\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.795899 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.795743 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6dfcb6ba-9dbd-4559-884e-a31a3989de13-metrics-cert\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.795899 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.795774 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6dfcb6ba-9dbd-4559-884e-a31a3989de13-manager-config\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.795899 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.795795 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lb96\" (UniqueName: \"kubernetes.io/projected/6dfcb6ba-9dbd-4559-884e-a31a3989de13-kube-api-access-4lb96\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.796428 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.796403 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6dfcb6ba-9dbd-4559-884e-a31a3989de13-manager-config\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.798042 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.798017 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6dfcb6ba-9dbd-4559-884e-a31a3989de13-cert\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.798131 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.798067 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6dfcb6ba-9dbd-4559-884e-a31a3989de13-metrics-cert\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.806659 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.806636 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lb96\" (UniqueName: \"kubernetes.io/projected/6dfcb6ba-9dbd-4559-884e-a31a3989de13-kube-api-access-4lb96\") pod \"lws-controller-manager-78958bf977-j5rxz\" (UID: \"6dfcb6ba-9dbd-4559-884e-a31a3989de13\") " pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:34.893589 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:34.893507 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:35.035162 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:35.035067 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz"] Jul 06 19:14:35.037577 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:14:35.037548 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfcb6ba_9dbd_4559_884e_a31a3989de13.slice/crio-c6cf5647585bada55941aeb6e311dd26d065a1a0df32a3aed11a9ca37abd0ed0 WatchSource:0}: Error finding container c6cf5647585bada55941aeb6e311dd26d065a1a0df32a3aed11a9ca37abd0ed0: Status 404 returned error can't find the container with id c6cf5647585bada55941aeb6e311dd26d065a1a0df32a3aed11a9ca37abd0ed0 Jul 06 19:14:36.013379 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.013337 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" event={"ID":"6dfcb6ba-9dbd-4559-884e-a31a3989de13","Type":"ContainerStarted","Data":"c6cf5647585bada55941aeb6e311dd26d065a1a0df32a3aed11a9ca37abd0ed0"} Jul 06 19:14:36.069558 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.069522 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr"] Jul 06 19:14:36.072665 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.072641 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.076312 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.075993 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:14:36.076312 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.076124 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:14:36.076312 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.076228 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wpwt9\"" Jul 06 19:14:36.076560 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.076411 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:14:36.076761 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.076726 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:14:36.088681 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.088629 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr"] Jul 06 19:14:36.111181 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.111142 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr9lp\" (UniqueName: \"kubernetes.io/projected/d32cff0c-e6d9-456a-b294-00d2d6db5fad-kube-api-access-nr9lp\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.111343 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.111221 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d32cff0c-e6d9-456a-b294-00d2d6db5fad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.111343 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.111268 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d32cff0c-e6d9-456a-b294-00d2d6db5fad-webhook-cert\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.111343 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.111296 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d32cff0c-e6d9-456a-b294-00d2d6db5fad-manifests\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.111343 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.111323 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d32cff0c-e6d9-456a-b294-00d2d6db5fad-tmp\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.211878 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.211836 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d32cff0c-e6d9-456a-b294-00d2d6db5fad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.212058 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.211897 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d32cff0c-e6d9-456a-b294-00d2d6db5fad-webhook-cert\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.212058 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.211999 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d32cff0c-e6d9-456a-b294-00d2d6db5fad-manifests\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.212058 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.212029 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d32cff0c-e6d9-456a-b294-00d2d6db5fad-tmp\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.212223 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.212112 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr9lp\" (UniqueName: \"kubernetes.io/projected/d32cff0c-e6d9-456a-b294-00d2d6db5fad-kube-api-access-nr9lp\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.212464 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.212435 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d32cff0c-e6d9-456a-b294-00d2d6db5fad-manifests\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.214794 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.214635 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d32cff0c-e6d9-456a-b294-00d2d6db5fad-tmp\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.214794 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.214675 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d32cff0c-e6d9-456a-b294-00d2d6db5fad-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.214794 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.214746 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d32cff0c-e6d9-456a-b294-00d2d6db5fad-webhook-cert\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.222406 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.222383 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr9lp\" (UniqueName: \"kubernetes.io/projected/d32cff0c-e6d9-456a-b294-00d2d6db5fad-kube-api-access-nr9lp\") pod \"opendatahub-operator-controller-manager-55c47dfb7f-vgtnr\" (UID: \"d32cff0c-e6d9-456a-b294-00d2d6db5fad\") " pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:36.373203 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.373162 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:14:36.377371 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.377346 2554 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:14:36.383264 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.383237 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:14:36.392227 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.392204 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wpwt9\"" Jul 06 19:14:36.400608 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:36.400583 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:37.486911 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:37.486887 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr"] Jul 06 19:14:37.489339 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:14:37.489316 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32cff0c_e6d9_456a_b294_00d2d6db5fad.slice/crio-d205d350280879b6ca1a51620e835c0d99798ff1a951f060299a0aab814e5e73 WatchSource:0}: Error finding container d205d350280879b6ca1a51620e835c0d99798ff1a951f060299a0aab814e5e73: Status 404 returned error can't find the container with id d205d350280879b6ca1a51620e835c0d99798ff1a951f060299a0aab814e5e73 Jul 06 19:14:38.025061 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:38.025016 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" event={"ID":"6dfcb6ba-9dbd-4559-884e-a31a3989de13","Type":"ContainerStarted","Data":"e33a4c5b402dfd4d8ea78279caf03269e73ae415a75d64cce9f680c97b3ccda3"} Jul 06 19:14:38.025253 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:38.025172 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:38.026831 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:38.026801 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" event={"ID":"d32cff0c-e6d9-456a-b294-00d2d6db5fad","Type":"ContainerStarted","Data":"d205d350280879b6ca1a51620e835c0d99798ff1a951f060299a0aab814e5e73"} Jul 06 19:14:38.054633 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:38.054571 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" podStartSLOduration=1.696395924 podStartE2EDuration="4.054550614s" totalImagesPullingTime="2.35815469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:14:38.048998213 +0000 UTC m=+303.532221787" watchObservedRunningTime="2026-07-06 19:14:38.054550614 +0000 UTC m=+303.537774188" Jul 06 19:14:42.049823 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:42.049726 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" event={"ID":"d32cff0c-e6d9-456a-b294-00d2d6db5fad","Type":"ContainerStarted","Data":"5944fb2c1c857da7507e9162c2df208a2af1fca567b70624ca1c0d597b645a53"} Jul 06 19:14:43.055240 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:43.055146 2554 generic.go:356] "Generic (PLEG): container finished" podID="d32cff0c-e6d9-456a-b294-00d2d6db5fad" containerID="5944fb2c1c857da7507e9162c2df208a2af1fca567b70624ca1c0d597b645a53" exitCode=0 Jul 06 19:14:43.055240 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:43.055226 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" event={"ID":"d32cff0c-e6d9-456a-b294-00d2d6db5fad","Type":"ContainerDied","Data":"5944fb2c1c857da7507e9162c2df208a2af1fca567b70624ca1c0d597b645a53"} Jul 06 19:14:43.055776 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:43.055762 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:14:44.061634 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:44.061595 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" event={"ID":"d32cff0c-e6d9-456a-b294-00d2d6db5fad","Type":"ContainerStarted","Data":"a79ff1db7b1bcca436529bcf584a8c30c308e35545a42399d0a1c6444a8b2832"} Jul 06 19:14:44.062128 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:44.061687 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:44.085880 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:44.085830 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" podStartSLOduration=2.547500664 podStartE2EDuration="8.085811202s" totalImagesPullingTime="4.497850931s" totalInitContainerRuntime="1.040459607s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:14:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:14:44.08410052 +0000 UTC m=+309.567324113" watchObservedRunningTime="2026-07-06 19:14:44.085811202 +0000 UTC m=+309.569034777" Jul 06 19:14:49.034094 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:49.034066 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78958bf977-j5rxz" Jul 06 19:14:55.068816 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:55.068783 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55c47dfb7f-vgtnr" Jul 06 19:14:57.174307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.174278 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-754576df6f-8mtfg"] Jul 06 19:14:57.177253 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.177232 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.180335 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.180310 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:14:57.180458 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.180406 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:14:57.180717 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.180698 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:14:57.181808 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.181792 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rdfhv\"" Jul 06 19:14:57.181808 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.181800 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:14:57.194714 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.194689 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-754576df6f-8mtfg"] Jul 06 19:14:57.302118 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.302078 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/126eac73-4dcc-4849-8744-8d42cfe8db14-tls-certs\") pod \"kube-auth-proxy-754576df6f-8mtfg\" (UID: \"126eac73-4dcc-4849-8744-8d42cfe8db14\") " pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.302317 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.302175 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdpg7\" (UniqueName: \"kubernetes.io/projected/126eac73-4dcc-4849-8744-8d42cfe8db14-kube-api-access-xdpg7\") pod \"kube-auth-proxy-754576df6f-8mtfg\" (UID: \"126eac73-4dcc-4849-8744-8d42cfe8db14\") " pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.302317 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.302231 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/126eac73-4dcc-4849-8744-8d42cfe8db14-tmp\") pod \"kube-auth-proxy-754576df6f-8mtfg\" (UID: \"126eac73-4dcc-4849-8744-8d42cfe8db14\") " pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.403089 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.403047 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/126eac73-4dcc-4849-8744-8d42cfe8db14-tls-certs\") pod \"kube-auth-proxy-754576df6f-8mtfg\" (UID: \"126eac73-4dcc-4849-8744-8d42cfe8db14\") " pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.403259 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.403129 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdpg7\" (UniqueName: \"kubernetes.io/projected/126eac73-4dcc-4849-8744-8d42cfe8db14-kube-api-access-xdpg7\") pod \"kube-auth-proxy-754576df6f-8mtfg\" (UID: \"126eac73-4dcc-4849-8744-8d42cfe8db14\") " pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.403259 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.403178 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/126eac73-4dcc-4849-8744-8d42cfe8db14-tmp\") pod \"kube-auth-proxy-754576df6f-8mtfg\" (UID: \"126eac73-4dcc-4849-8744-8d42cfe8db14\") " pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.405571 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.405539 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/126eac73-4dcc-4849-8744-8d42cfe8db14-tmp\") pod \"kube-auth-proxy-754576df6f-8mtfg\" (UID: \"126eac73-4dcc-4849-8744-8d42cfe8db14\") " pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.405987 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.405967 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/126eac73-4dcc-4849-8744-8d42cfe8db14-tls-certs\") pod \"kube-auth-proxy-754576df6f-8mtfg\" (UID: \"126eac73-4dcc-4849-8744-8d42cfe8db14\") " pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.420218 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.420196 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdpg7\" (UniqueName: \"kubernetes.io/projected/126eac73-4dcc-4849-8744-8d42cfe8db14-kube-api-access-xdpg7\") pod \"kube-auth-proxy-754576df6f-8mtfg\" (UID: \"126eac73-4dcc-4849-8744-8d42cfe8db14\") " pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.487154 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.487066 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" Jul 06 19:14:57.633878 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:57.633847 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-754576df6f-8mtfg"] Jul 06 19:14:57.637104 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:14:57.637074 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126eac73_4dcc_4849_8744_8d42cfe8db14.slice/crio-252356ccfcbcfe3f34c3eebd2b3ca03eaafe7dca45ac42553ba5af4b08115c2f WatchSource:0}: Error finding container 252356ccfcbcfe3f34c3eebd2b3ca03eaafe7dca45ac42553ba5af4b08115c2f: Status 404 returned error can't find the container with id 252356ccfcbcfe3f34c3eebd2b3ca03eaafe7dca45ac42553ba5af4b08115c2f Jul 06 19:14:58.125236 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:14:58.125205 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" event={"ID":"126eac73-4dcc-4849-8744-8d42cfe8db14","Type":"ContainerStarted","Data":"252356ccfcbcfe3f34c3eebd2b3ca03eaafe7dca45ac42553ba5af4b08115c2f"} Jul 06 19:15:03.158809 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:03.158769 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" event={"ID":"126eac73-4dcc-4849-8744-8d42cfe8db14","Type":"ContainerStarted","Data":"a57d413cccf1cd6de167b82e712bea028085309216f8b18b53695fcafa670cad"} Jul 06 19:15:03.190089 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:03.190037 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-754576df6f-8mtfg" podStartSLOduration=1.175749595 podStartE2EDuration="6.190021763s" totalImagesPullingTime="5.014272168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:03.188651346 +0000 UTC m=+328.671874920" watchObservedRunningTime="2026-07-06 19:15:03.190021763 +0000 UTC m=+328.673245336" Jul 06 19:15:07.301512 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.301477 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg"] Jul 06 19:15:07.303781 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.303765 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" Jul 06 19:15:07.306754 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.306709 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 19:15:07.308275 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.308252 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4nrml\"" Jul 06 19:15:07.317286 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.317263 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg"] Jul 06 19:15:07.394307 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.394271 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d5ae65-b3cb-41ec-859e-1ec405a2e877-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dcwfg\" (UID: \"85d5ae65-b3cb-41ec-859e-1ec405a2e877\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" Jul 06 19:15:07.394460 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.394329 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crcq2\" (UniqueName: \"kubernetes.io/projected/85d5ae65-b3cb-41ec-859e-1ec405a2e877-kube-api-access-crcq2\") pod \"llmisvc-controller-manager-777c6bd7d5-dcwfg\" (UID: \"85d5ae65-b3cb-41ec-859e-1ec405a2e877\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" Jul 06 19:15:07.495492 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.495457 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crcq2\" (UniqueName: \"kubernetes.io/projected/85d5ae65-b3cb-41ec-859e-1ec405a2e877-kube-api-access-crcq2\") pod \"llmisvc-controller-manager-777c6bd7d5-dcwfg\" (UID: \"85d5ae65-b3cb-41ec-859e-1ec405a2e877\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" Jul 06 19:15:07.495679 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.495558 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d5ae65-b3cb-41ec-859e-1ec405a2e877-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dcwfg\" (UID: \"85d5ae65-b3cb-41ec-859e-1ec405a2e877\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" Jul 06 19:15:07.498132 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.498099 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85d5ae65-b3cb-41ec-859e-1ec405a2e877-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-dcwfg\" (UID: \"85d5ae65-b3cb-41ec-859e-1ec405a2e877\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" Jul 06 19:15:07.510390 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.510368 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crcq2\" (UniqueName: \"kubernetes.io/projected/85d5ae65-b3cb-41ec-859e-1ec405a2e877-kube-api-access-crcq2\") pod \"llmisvc-controller-manager-777c6bd7d5-dcwfg\" (UID: \"85d5ae65-b3cb-41ec-859e-1ec405a2e877\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" Jul 06 19:15:07.614461 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.614429 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" Jul 06 19:15:07.746059 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:07.746033 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg"] Jul 06 19:15:07.748600 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:15:07.748570 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod85d5ae65_b3cb_41ec_859e_1ec405a2e877.slice/crio-af83ba70a5b2cb6e4076489ccf3d61380f4ccd6d1235875118cf1639d7472563 WatchSource:0}: Error finding container af83ba70a5b2cb6e4076489ccf3d61380f4ccd6d1235875118cf1639d7472563: Status 404 returned error can't find the container with id af83ba70a5b2cb6e4076489ccf3d61380f4ccd6d1235875118cf1639d7472563 Jul 06 19:15:08.182157 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:08.182119 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" event={"ID":"85d5ae65-b3cb-41ec-859e-1ec405a2e877","Type":"ContainerStarted","Data":"af83ba70a5b2cb6e4076489ccf3d61380f4ccd6d1235875118cf1639d7472563"} Jul 06 19:15:12.205072 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:12.205036 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" event={"ID":"85d5ae65-b3cb-41ec-859e-1ec405a2e877","Type":"ContainerStarted","Data":"43c957c37aa8066e593872e067e130d03cb4761265aeb3b92268ccef972a02be"} Jul 06 19:15:12.205567 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:12.205150 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" Jul 06 19:15:43.212583 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:43.212508 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" Jul 06 19:15:43.229847 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:15:43.229797 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-dcwfg" podStartSLOduration=32.503216402 podStartE2EDuration="36.229782237s" totalImagesPullingTime="3.726565835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:12.243711368 +0000 UTC m=+337.726934942" watchObservedRunningTime="2026-07-06 19:15:43.229782237 +0000 UTC m=+368.713005809" Jul 06 19:16:53.727964 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.727930 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh"] Jul 06 19:16:53.731181 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.731164 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:53.734237 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.734220 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4n949\"" Jul 06 19:16:53.734365 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.734220 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:16:53.734365 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.734344 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:16:53.754191 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.754165 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh"] Jul 06 19:16:53.794700 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.794669 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-b4wzh\" (UID: \"6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:53.795078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.794716 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qpk\" (UniqueName: \"kubernetes.io/projected/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-kube-api-access-s5qpk\") pod \"kuadrant-operator-controller-manager-55467d9f7-b4wzh\" (UID: \"6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:53.895837 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.895800 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-b4wzh\" (UID: \"6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:53.896018 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.895844 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qpk\" (UniqueName: \"kubernetes.io/projected/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-kube-api-access-s5qpk\") pod \"kuadrant-operator-controller-manager-55467d9f7-b4wzh\" (UID: \"6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:53.896202 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.896176 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-b4wzh\" (UID: \"6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:53.904946 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:53.904913 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qpk\" (UniqueName: \"kubernetes.io/projected/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-kube-api-access-s5qpk\") pod \"kuadrant-operator-controller-manager-55467d9f7-b4wzh\" (UID: \"6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:54.009168 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.009079 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh"] Jul 06 19:16:54.009371 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.009360 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:54.015978 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.015955 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh"] Jul 06 19:16:54.031624 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.031593 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8"] Jul 06 19:16:54.035113 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.035091 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:16:54.048475 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.048444 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8"] Jul 06 19:16:54.098566 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.098532 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dba11cc5-ca63-41eb-82ec-a9974ec6415c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tt5n8\" (UID: \"dba11cc5-ca63-41eb-82ec-a9974ec6415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:16:54.098743 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.098583 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5ht8\" (UniqueName: \"kubernetes.io/projected/dba11cc5-ca63-41eb-82ec-a9974ec6415c-kube-api-access-c5ht8\") pod \"kuadrant-operator-controller-manager-55467d9f7-tt5n8\" (UID: \"dba11cc5-ca63-41eb-82ec-a9974ec6415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:16:54.199725 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.199679 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dba11cc5-ca63-41eb-82ec-a9974ec6415c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tt5n8\" (UID: \"dba11cc5-ca63-41eb-82ec-a9974ec6415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:16:54.199926 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.199762 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5ht8\" (UniqueName: \"kubernetes.io/projected/dba11cc5-ca63-41eb-82ec-a9974ec6415c-kube-api-access-c5ht8\") pod \"kuadrant-operator-controller-manager-55467d9f7-tt5n8\" (UID: \"dba11cc5-ca63-41eb-82ec-a9974ec6415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:16:54.200114 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.200092 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dba11cc5-ca63-41eb-82ec-a9974ec6415c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-tt5n8\" (UID: \"dba11cc5-ca63-41eb-82ec-a9974ec6415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:16:54.208811 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.208783 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5ht8\" (UniqueName: \"kubernetes.io/projected/dba11cc5-ca63-41eb-82ec-a9974ec6415c-kube-api-access-c5ht8\") pod \"kuadrant-operator-controller-manager-55467d9f7-tt5n8\" (UID: \"dba11cc5-ca63-41eb-82ec-a9974ec6415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:16:54.347358 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.347325 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:16:54.507404 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.507378 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8"] Jul 06 19:16:54.510471 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:16:54.510444 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba11cc5_ca63_41eb_82ec_a9974ec6415c.slice/crio-a165e35905b0abde3faf07bc12ce0ad865bacda5e787dbd2ffa03d43bacee4c3 WatchSource:0}: Error finding container a165e35905b0abde3faf07bc12ce0ad865bacda5e787dbd2ffa03d43bacee4c3: Status 404 returned error can't find the container with id a165e35905b0abde3faf07bc12ce0ad865bacda5e787dbd2ffa03d43bacee4c3 Jul 06 19:16:54.645651 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:54.645565 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" event={"ID":"dba11cc5-ca63-41eb-82ec-a9974ec6415c","Type":"ContainerStarted","Data":"a165e35905b0abde3faf07bc12ce0ad865bacda5e787dbd2ffa03d43bacee4c3"} Jul 06 19:16:56.556641 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:16:56.556588 2554 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-b4wzh_kuadrant-system_6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9_0(8917f5a25bb513ffd7858bf5c4831e7c0942703339c6f2a17916297b22b6e51d): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-b4wzh to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 06 19:16:56.557013 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:16:56.556665 2554 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-b4wzh_kuadrant-system_6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9_0(8917f5a25bb513ffd7858bf5c4831e7c0942703339c6f2a17916297b22b6e51d): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-b4wzh to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:56.661552 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:56.661520 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:56.664166 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:16:56.664133 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-b4wzh\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:56.669400 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:56.669381 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:56.671708 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:16:56.671684 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-b4wzh\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:56.723174 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:56.723137 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5qpk\" (UniqueName: \"kubernetes.io/projected/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-kube-api-access-s5qpk\") pod \"6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9\" (UID: \"6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9\") " Jul 06 19:16:56.723367 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:56.723195 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-extensions-socket-volume\") pod \"6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9\" (UID: \"6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9\") " Jul 06 19:16:56.723529 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:56.723501 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9" (UID: "6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:16:56.725709 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:56.725680 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-kube-api-access-s5qpk" (OuterVolumeSpecName: "kube-api-access-s5qpk") pod "6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9" (UID: "6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9"). InnerVolumeSpecName "kube-api-access-s5qpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:16:56.824079 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:56.824050 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5qpk\" (UniqueName: \"kubernetes.io/projected/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-kube-api-access-s5qpk\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:16:56.824079 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:56.824077 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9-extensions-socket-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:16:57.666136 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:57.666103 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:57.669921 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:16:57.669887 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-b4wzh\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:57.674658 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:16:57.674632 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-b4wzh\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-b4wzh" Jul 06 19:16:58.530204 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:16:58.530172 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9" path="/var/lib/kubelet/pods/6c4d4dbe-cbea-4fb7-88da-c4301d2fc7c9/volumes" Jul 06 19:17:00.682936 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:00.682900 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" event={"ID":"dba11cc5-ca63-41eb-82ec-a9974ec6415c","Type":"ContainerStarted","Data":"fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba"} Jul 06 19:17:00.683331 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:00.683030 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:17:00.711017 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:00.710965 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" podStartSLOduration=1.157334142 podStartE2EDuration="6.71094947s" totalImagesPullingTime="5.553615328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:17:00.708608692 +0000 UTC m=+446.191832267" watchObservedRunningTime="2026-07-06 19:17:00.71094947 +0000 UTC m=+446.194173042" Jul 06 19:17:11.689943 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:11.689908 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:17:24.450298 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.450263 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8"] Jul 06 19:17:24.450755 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.450459 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" podUID="dba11cc5-ca63-41eb-82ec-a9974ec6415c" containerName="manager" containerID="cri-o://fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba" gracePeriod=10 Jul 06 19:17:24.701128 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.701059 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:17:24.773159 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.773131 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5ht8\" (UniqueName: \"kubernetes.io/projected/dba11cc5-ca63-41eb-82ec-a9974ec6415c-kube-api-access-c5ht8\") pod \"dba11cc5-ca63-41eb-82ec-a9974ec6415c\" (UID: \"dba11cc5-ca63-41eb-82ec-a9974ec6415c\") " Jul 06 19:17:24.773323 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.773230 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dba11cc5-ca63-41eb-82ec-a9974ec6415c-extensions-socket-volume\") pod \"dba11cc5-ca63-41eb-82ec-a9974ec6415c\" (UID: \"dba11cc5-ca63-41eb-82ec-a9974ec6415c\") " Jul 06 19:17:24.773756 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.773702 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dba11cc5-ca63-41eb-82ec-a9974ec6415c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dba11cc5-ca63-41eb-82ec-a9974ec6415c" (UID: "dba11cc5-ca63-41eb-82ec-a9974ec6415c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:17:24.775423 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.775398 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba11cc5-ca63-41eb-82ec-a9974ec6415c-kube-api-access-c5ht8" (OuterVolumeSpecName: "kube-api-access-c5ht8") pod "dba11cc5-ca63-41eb-82ec-a9974ec6415c" (UID: "dba11cc5-ca63-41eb-82ec-a9974ec6415c"). InnerVolumeSpecName "kube-api-access-c5ht8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:17:24.787977 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.787948 2554 generic.go:356] "Generic (PLEG): container finished" podID="dba11cc5-ca63-41eb-82ec-a9974ec6415c" containerID="fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba" exitCode=0 Jul 06 19:17:24.788071 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.787997 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" Jul 06 19:17:24.788071 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.788015 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" event={"ID":"dba11cc5-ca63-41eb-82ec-a9974ec6415c","Type":"ContainerDied","Data":"fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba"} Jul 06 19:17:24.788071 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.788057 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8" event={"ID":"dba11cc5-ca63-41eb-82ec-a9974ec6415c","Type":"ContainerDied","Data":"a165e35905b0abde3faf07bc12ce0ad865bacda5e787dbd2ffa03d43bacee4c3"} Jul 06 19:17:24.788224 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.788078 2554 scope.go:117] "RemoveContainer" containerID="fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba" Jul 06 19:17:24.798711 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.798686 2554 scope.go:117] "RemoveContainer" containerID="fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba" Jul 06 19:17:24.799217 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:17:24.799193 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba\": container with ID starting with fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba not found: ID does not exist" containerID="fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba" Jul 06 19:17:24.799372 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.799224 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba"} err="failed to get container status \"fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba\": rpc error: code = NotFound desc = could not find container \"fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba\": container with ID starting with fee91b76b7ce3a91760f9588bd3e11bb33147d1b6336841c9798798e20d37cba not found: ID does not exist" Jul 06 19:17:24.809617 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.809596 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8"] Jul 06 19:17:24.822117 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.822085 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-tt5n8"] Jul 06 19:17:24.873847 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.873819 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dba11cc5-ca63-41eb-82ec-a9974ec6415c-extensions-socket-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:17:24.873847 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:24.873841 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5ht8\" (UniqueName: \"kubernetes.io/projected/dba11cc5-ca63-41eb-82ec-a9974ec6415c-kube-api-access-c5ht8\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:17:26.529720 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:26.529690 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba11cc5-ca63-41eb-82ec-a9974ec6415c" path="/var/lib/kubelet/pods/dba11cc5-ca63-41eb-82ec-a9974ec6415c/volumes" Jul 06 19:17:36.304841 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.304805 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74ff6b4789-sjf5k"] Jul 06 19:17:36.305232 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.305173 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba11cc5-ca63-41eb-82ec-a9974ec6415c" containerName="manager" Jul 06 19:17:36.305232 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.305184 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba11cc5-ca63-41eb-82ec-a9974ec6415c" containerName="manager" Jul 06 19:17:36.305299 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.305239 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba11cc5-ca63-41eb-82ec-a9974ec6415c" containerName="manager" Jul 06 19:17:36.307301 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.307282 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.319563 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.319532 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74ff6b4789-sjf5k"] Jul 06 19:17:36.376649 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.376618 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-oauth-serving-cert\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.376649 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.376656 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-service-ca\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.376877 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.376679 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-trusted-ca-bundle\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.376877 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.376766 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f5c83ef-00b1-463c-9622-a0641cad5d08-console-serving-cert\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.376877 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.376815 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-console-config\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.376976 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.376879 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f5c83ef-00b1-463c-9622-a0641cad5d08-console-oauth-config\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.376976 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.376896 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcsb\" (UniqueName: \"kubernetes.io/projected/7f5c83ef-00b1-463c-9622-a0641cad5d08-kube-api-access-5xcsb\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.478060 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.478018 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-service-ca\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.478060 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.478060 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-trusted-ca-bundle\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.478300 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.478084 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f5c83ef-00b1-463c-9622-a0641cad5d08-console-serving-cert\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.478300 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.478106 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-console-config\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.478300 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.478177 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f5c83ef-00b1-463c-9622-a0641cad5d08-console-oauth-config\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.478300 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.478202 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcsb\" (UniqueName: \"kubernetes.io/projected/7f5c83ef-00b1-463c-9622-a0641cad5d08-kube-api-access-5xcsb\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.478300 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.478241 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-oauth-serving-cert\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.478942 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.478917 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-console-config\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.479055 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.478940 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-service-ca\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.479055 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.479014 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-trusted-ca-bundle\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.479055 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.479035 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f5c83ef-00b1-463c-9622-a0641cad5d08-oauth-serving-cert\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.480686 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.480657 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f5c83ef-00b1-463c-9622-a0641cad5d08-console-oauth-config\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.480807 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.480792 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f5c83ef-00b1-463c-9622-a0641cad5d08-console-serving-cert\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.495840 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.495816 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcsb\" (UniqueName: \"kubernetes.io/projected/7f5c83ef-00b1-463c-9622-a0641cad5d08-kube-api-access-5xcsb\") pod \"console-74ff6b4789-sjf5k\" (UID: \"7f5c83ef-00b1-463c-9622-a0641cad5d08\") " pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.616282 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.616247 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:36.743211 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.743184 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74ff6b4789-sjf5k"] Jul 06 19:17:36.745562 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:17:36.745537 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5c83ef_00b1_463c_9622_a0641cad5d08.slice/crio-755be4d9cf2f7b68a1fd81a6a995fab31286efd811abf26d70c5112e69cd0399 WatchSource:0}: Error finding container 755be4d9cf2f7b68a1fd81a6a995fab31286efd811abf26d70c5112e69cd0399: Status 404 returned error can't find the container with id 755be4d9cf2f7b68a1fd81a6a995fab31286efd811abf26d70c5112e69cd0399 Jul 06 19:17:36.852592 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.852557 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74ff6b4789-sjf5k" event={"ID":"7f5c83ef-00b1-463c-9622-a0641cad5d08","Type":"ContainerStarted","Data":"b8d842eafc3a17ba2788f61cec09f56e9f3e5bb5ff79f50117b7ea490539f14e"} Jul 06 19:17:36.852780 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.852634 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74ff6b4789-sjf5k" event={"ID":"7f5c83ef-00b1-463c-9622-a0641cad5d08","Type":"ContainerStarted","Data":"755be4d9cf2f7b68a1fd81a6a995fab31286efd811abf26d70c5112e69cd0399"} Jul 06 19:17:36.868595 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:36.868339 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74ff6b4789-sjf5k" podStartSLOduration=0.868318233 podStartE2EDuration="868.318233ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:17:36.868247646 +0000 UTC m=+482.351471220" watchObservedRunningTime="2026-07-06 19:17:36.868318233 +0000 UTC m=+482.351541808" Jul 06 19:17:44.192663 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.192622 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:17:44.196617 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.196584 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" Jul 06 19:17:44.200154 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.200129 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:17:44.201075 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.200911 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:17:44.201075 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.200927 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bdxch\"" Jul 06 19:17:44.202273 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.202247 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:17:44.204308 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.204285 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:17:44.237399 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.237369 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:17:44.347797 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.347764 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krq5x\" (UniqueName: \"kubernetes.io/projected/ff816c7b-3f56-4364-8419-d4d11089abcc-kube-api-access-krq5x\") pod \"limitador-limitador-54d6d988d5-7jjw8\" (UID: \"ff816c7b-3f56-4364-8419-d4d11089abcc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" Jul 06 19:17:44.348011 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.347868 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ff816c7b-3f56-4364-8419-d4d11089abcc-config-file\") pod \"limitador-limitador-54d6d988d5-7jjw8\" (UID: \"ff816c7b-3f56-4364-8419-d4d11089abcc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" Jul 06 19:17:44.449236 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.449137 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krq5x\" (UniqueName: \"kubernetes.io/projected/ff816c7b-3f56-4364-8419-d4d11089abcc-kube-api-access-krq5x\") pod \"limitador-limitador-54d6d988d5-7jjw8\" (UID: \"ff816c7b-3f56-4364-8419-d4d11089abcc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" Jul 06 19:17:44.449236 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.449207 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ff816c7b-3f56-4364-8419-d4d11089abcc-config-file\") pod \"limitador-limitador-54d6d988d5-7jjw8\" (UID: \"ff816c7b-3f56-4364-8419-d4d11089abcc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" Jul 06 19:17:44.449887 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.449864 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ff816c7b-3f56-4364-8419-d4d11089abcc-config-file\") pod \"limitador-limitador-54d6d988d5-7jjw8\" (UID: \"ff816c7b-3f56-4364-8419-d4d11089abcc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" Jul 06 19:17:44.459695 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.459673 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krq5x\" (UniqueName: \"kubernetes.io/projected/ff816c7b-3f56-4364-8419-d4d11089abcc-kube-api-access-krq5x\") pod \"limitador-limitador-54d6d988d5-7jjw8\" (UID: \"ff816c7b-3f56-4364-8419-d4d11089abcc\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" Jul 06 19:17:44.511302 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.511268 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" Jul 06 19:17:44.645712 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.645688 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:17:44.648184 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:17:44.648153 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff816c7b_3f56_4364_8419_d4d11089abcc.slice/crio-08fa6d75c89a7175fe1787862e32c224e9db82e8b5cf2a66798831daf53c8529 WatchSource:0}: Error finding container 08fa6d75c89a7175fe1787862e32c224e9db82e8b5cf2a66798831daf53c8529: Status 404 returned error can't find the container with id 08fa6d75c89a7175fe1787862e32c224e9db82e8b5cf2a66798831daf53c8529 Jul 06 19:17:44.889034 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:44.889001 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" event={"ID":"ff816c7b-3f56-4364-8419-d4d11089abcc","Type":"ContainerStarted","Data":"08fa6d75c89a7175fe1787862e32c224e9db82e8b5cf2a66798831daf53c8529"} Jul 06 19:17:46.617230 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:46.617195 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:46.617703 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:46.617257 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:46.623702 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:46.623678 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:46.903867 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:46.903777 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74ff6b4789-sjf5k" Jul 06 19:17:46.947111 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:46.947004 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4c6fb4cc-t89wk"] Jul 06 19:17:47.906555 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:47.906519 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" event={"ID":"ff816c7b-3f56-4364-8419-d4d11089abcc","Type":"ContainerStarted","Data":"3d933959531c64357f9e632f433593f8e572a05e1bf2531a65220fa50e6de66b"} Jul 06 19:17:47.906967 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:47.906835 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" Jul 06 19:17:47.924440 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:47.924391 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" podStartSLOduration=1.332930369 podStartE2EDuration="3.924376857s" totalImagesPullingTime="2.591446488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:17:47.922929347 +0000 UTC m=+493.406152921" watchObservedRunningTime="2026-07-06 19:17:47.924376857 +0000 UTC m=+493.407600431" Jul 06 19:17:58.913379 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:17:58.913348 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7jjw8" Jul 06 19:18:11.974801 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:11.974708 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6c4c6fb4cc-t89wk" podUID="d72e8db6-5f2d-4bd3-9007-e92c18ff0094" containerName="console" containerID="cri-o://d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40" gracePeriod=15 Jul 06 19:18:12.219417 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.219388 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:18:12.283466 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.283372 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-oauth-serving-cert\") pod \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " Jul 06 19:18:12.283466 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.283447 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q76ng\" (UniqueName: \"kubernetes.io/projected/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-kube-api-access-q76ng\") pod \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " Jul 06 19:18:12.283659 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.283499 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-service-ca\") pod \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " Jul 06 19:18:12.283659 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.283525 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-oauth-config\") pod \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " Jul 06 19:18:12.283765 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.283651 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-serving-cert\") pod \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " Jul 06 19:18:12.283765 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.283700 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-config\") pod \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " Jul 06 19:18:12.283903 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.283796 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-trusted-ca-bundle\") pod \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\" (UID: \"d72e8db6-5f2d-4bd3-9007-e92c18ff0094\") " Jul 06 19:18:12.283961 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.283907 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d72e8db6-5f2d-4bd3-9007-e92c18ff0094" (UID: "d72e8db6-5f2d-4bd3-9007-e92c18ff0094"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:12.284066 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.284029 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-service-ca" (OuterVolumeSpecName: "service-ca") pod "d72e8db6-5f2d-4bd3-9007-e92c18ff0094" (UID: "d72e8db6-5f2d-4bd3-9007-e92c18ff0094"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:12.284179 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.284152 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-config" (OuterVolumeSpecName: "console-config") pod "d72e8db6-5f2d-4bd3-9007-e92c18ff0094" (UID: "d72e8db6-5f2d-4bd3-9007-e92c18ff0094"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:12.284263 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.284247 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-oauth-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:18:12.284313 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.284255 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d72e8db6-5f2d-4bd3-9007-e92c18ff0094" (UID: "d72e8db6-5f2d-4bd3-9007-e92c18ff0094"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:18:12.284313 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.284269 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:18:12.284313 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.284283 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:18:12.285719 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.285693 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-kube-api-access-q76ng" (OuterVolumeSpecName: "kube-api-access-q76ng") pod "d72e8db6-5f2d-4bd3-9007-e92c18ff0094" (UID: "d72e8db6-5f2d-4bd3-9007-e92c18ff0094"). InnerVolumeSpecName "kube-api-access-q76ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:18:12.286178 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.286146 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d72e8db6-5f2d-4bd3-9007-e92c18ff0094" (UID: "d72e8db6-5f2d-4bd3-9007-e92c18ff0094"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:12.286178 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.286167 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d72e8db6-5f2d-4bd3-9007-e92c18ff0094" (UID: "d72e8db6-5f2d-4bd3-9007-e92c18ff0094"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:18:12.384693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.384649 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-oauth-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:18:12.384693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.384682 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-console-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:18:12.384693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.384691 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-trusted-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:18:12.384693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:12.384699 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q76ng\" (UniqueName: \"kubernetes.io/projected/d72e8db6-5f2d-4bd3-9007-e92c18ff0094-kube-api-access-q76ng\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:18:13.027935 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:13.027902 2554 generic.go:356] "Generic (PLEG): container finished" podID="d72e8db6-5f2d-4bd3-9007-e92c18ff0094" containerID="d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40" exitCode=0 Jul 06 19:18:13.028351 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:13.028000 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4c6fb4cc-t89wk" Jul 06 19:18:13.028351 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:13.028023 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4c6fb4cc-t89wk" event={"ID":"d72e8db6-5f2d-4bd3-9007-e92c18ff0094","Type":"ContainerDied","Data":"d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40"} Jul 06 19:18:13.028351 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:13.028069 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4c6fb4cc-t89wk" event={"ID":"d72e8db6-5f2d-4bd3-9007-e92c18ff0094","Type":"ContainerDied","Data":"c71f21c72ab9b8d69a10d8b80aa89b041c812c654684dfed4c4616711af5ed49"} Jul 06 19:18:13.028351 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:13.028090 2554 scope.go:117] "RemoveContainer" containerID="d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40" Jul 06 19:18:13.036332 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:13.036314 2554 scope.go:117] "RemoveContainer" containerID="d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40" Jul 06 19:18:13.036584 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:18:13.036565 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40\": container with ID starting with d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40 not found: ID does not exist" containerID="d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40" Jul 06 19:18:13.036628 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:13.036593 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40"} err="failed to get container status \"d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40\": rpc error: code = NotFound desc = could not find container \"d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40\": container with ID starting with d55603f7abe005224b732fd7a38563e4382500e34b0e119ce888bf9c97db2b40 not found: ID does not exist" Jul 06 19:18:13.046134 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:13.046109 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4c6fb4cc-t89wk"] Jul 06 19:18:13.049966 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:13.049943 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c4c6fb4cc-t89wk"] Jul 06 19:18:14.530352 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:14.530272 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d72e8db6-5f2d-4bd3-9007-e92c18ff0094" path="/var/lib/kubelet/pods/d72e8db6-5f2d-4bd3-9007-e92c18ff0094/volumes" Jul 06 19:18:16.049812 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.049773 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tmzhp"] Jul 06 19:18:16.050319 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.050300 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d72e8db6-5f2d-4bd3-9007-e92c18ff0094" containerName="console" Jul 06 19:18:16.050396 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.050323 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d72e8db6-5f2d-4bd3-9007-e92c18ff0094" containerName="console" Jul 06 19:18:16.050451 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.050442 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d72e8db6-5f2d-4bd3-9007-e92c18ff0094" containerName="console" Jul 06 19:18:16.055070 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.055048 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tmzhp" Jul 06 19:18:16.057929 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.057908 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7q8l6\"" Jul 06 19:18:16.061318 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.061292 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tmzhp"] Jul 06 19:18:16.118354 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.118320 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jbj\" (UniqueName: \"kubernetes.io/projected/4f78b3da-5810-46f2-b875-d7abbc62f942-kube-api-access-c9jbj\") pod \"authorino-5d4b67c446-tmzhp\" (UID: \"4f78b3da-5810-46f2-b875-d7abbc62f942\") " pod="kuadrant-system/authorino-5d4b67c446-tmzhp" Jul 06 19:18:16.218720 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.218677 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jbj\" (UniqueName: \"kubernetes.io/projected/4f78b3da-5810-46f2-b875-d7abbc62f942-kube-api-access-c9jbj\") pod \"authorino-5d4b67c446-tmzhp\" (UID: \"4f78b3da-5810-46f2-b875-d7abbc62f942\") " pod="kuadrant-system/authorino-5d4b67c446-tmzhp" Jul 06 19:18:16.229061 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.229025 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jbj\" (UniqueName: \"kubernetes.io/projected/4f78b3da-5810-46f2-b875-d7abbc62f942-kube-api-access-c9jbj\") pod \"authorino-5d4b67c446-tmzhp\" (UID: \"4f78b3da-5810-46f2-b875-d7abbc62f942\") " pod="kuadrant-system/authorino-5d4b67c446-tmzhp" Jul 06 19:18:16.291601 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.291559 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tmzhp"] Jul 06 19:18:16.291911 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.291893 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tmzhp" Jul 06 19:18:16.433784 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:16.433757 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tmzhp"] Jul 06 19:18:16.436430 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:18:16.436404 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f78b3da_5810_46f2_b875_d7abbc62f942.slice/crio-10011f7e81815e707b79ed0e926065f8e3d5b76933cdcd490553ef0b25b65463 WatchSource:0}: Error finding container 10011f7e81815e707b79ed0e926065f8e3d5b76933cdcd490553ef0b25b65463: Status 404 returned error can't find the container with id 10011f7e81815e707b79ed0e926065f8e3d5b76933cdcd490553ef0b25b65463 Jul 06 19:18:17.048135 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:17.048099 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tmzhp" event={"ID":"4f78b3da-5810-46f2-b875-d7abbc62f942","Type":"ContainerStarted","Data":"10011f7e81815e707b79ed0e926065f8e3d5b76933cdcd490553ef0b25b65463"} Jul 06 19:18:21.069851 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:21.069816 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tmzhp" event={"ID":"4f78b3da-5810-46f2-b875-d7abbc62f942","Type":"ContainerStarted","Data":"089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0"} Jul 06 19:18:21.070283 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:21.069853 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-tmzhp" podUID="4f78b3da-5810-46f2-b875-d7abbc62f942" containerName="authorino" containerID="cri-o://089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0" gracePeriod=30 Jul 06 19:18:21.085532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:21.085486 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-tmzhp" podStartSLOduration=1.143381596 podStartE2EDuration="5.085469603s" totalImagesPullingTime="3.942088007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:18:21.084480664 +0000 UTC m=+526.567704249" watchObservedRunningTime="2026-07-06 19:18:21.085469603 +0000 UTC m=+526.568693176" Jul 06 19:18:21.344618 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:21.344545 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tmzhp" Jul 06 19:18:21.467500 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:21.467465 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9jbj\" (UniqueName: \"kubernetes.io/projected/4f78b3da-5810-46f2-b875-d7abbc62f942-kube-api-access-c9jbj\") pod \"4f78b3da-5810-46f2-b875-d7abbc62f942\" (UID: \"4f78b3da-5810-46f2-b875-d7abbc62f942\") " Jul 06 19:18:21.469597 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:21.469573 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f78b3da-5810-46f2-b875-d7abbc62f942-kube-api-access-c9jbj" (OuterVolumeSpecName: "kube-api-access-c9jbj") pod "4f78b3da-5810-46f2-b875-d7abbc62f942" (UID: "4f78b3da-5810-46f2-b875-d7abbc62f942"). InnerVolumeSpecName "kube-api-access-c9jbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:18:21.568850 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:21.568809 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jbj\" (UniqueName: \"kubernetes.io/projected/4f78b3da-5810-46f2-b875-d7abbc62f942-kube-api-access-c9jbj\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:18:22.075965 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:22.075924 2554 generic.go:356] "Generic (PLEG): container finished" podID="4f78b3da-5810-46f2-b875-d7abbc62f942" containerID="089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0" exitCode=0 Jul 06 19:18:22.076396 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:22.075973 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-tmzhp" Jul 06 19:18:22.076396 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:22.076007 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tmzhp" event={"ID":"4f78b3da-5810-46f2-b875-d7abbc62f942","Type":"ContainerDied","Data":"089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0"} Jul 06 19:18:22.076396 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:22.076050 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-tmzhp" event={"ID":"4f78b3da-5810-46f2-b875-d7abbc62f942","Type":"ContainerDied","Data":"10011f7e81815e707b79ed0e926065f8e3d5b76933cdcd490553ef0b25b65463"} Jul 06 19:18:22.076396 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:22.076067 2554 scope.go:117] "RemoveContainer" containerID="089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0" Jul 06 19:18:22.084638 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:22.084624 2554 scope.go:117] "RemoveContainer" containerID="089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0" Jul 06 19:18:22.084901 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:18:22.084880 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0\": container with ID starting with 089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0 not found: ID does not exist" containerID="089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0" Jul 06 19:18:22.084951 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:22.084911 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0"} err="failed to get container status \"089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0\": rpc error: code = NotFound desc = could not find container \"089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0\": container with ID starting with 089ace93971ec2c00e63e9c9e2976dfb9f029625e57d4e6a59b9ac8e12d4f9e0 not found: ID does not exist" Jul 06 19:18:22.097933 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:22.097911 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tmzhp"] Jul 06 19:18:22.101692 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:22.101670 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-tmzhp"] Jul 06 19:18:22.529766 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:22.529659 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f78b3da-5810-46f2-b875-d7abbc62f942" path="/var/lib/kubelet/pods/4f78b3da-5810-46f2-b875-d7abbc62f942/volumes" Jul 06 19:18:58.560908 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:18:58.560864 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:19:14.990486 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:14.990447 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:19:23.092003 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.091961 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv"] Jul 06 19:19:23.092394 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.092345 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f78b3da-5810-46f2-b875-d7abbc62f942" containerName="authorino" Jul 06 19:19:23.092394 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.092356 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f78b3da-5810-46f2-b875-d7abbc62f942" containerName="authorino" Jul 06 19:19:23.092465 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.092414 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f78b3da-5810-46f2-b875-d7abbc62f942" containerName="authorino" Jul 06 19:19:23.094912 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.094891 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.097814 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.097795 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:19:23.099104 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.099087 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 19:19:23.099151 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.099130 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mwzf5\"" Jul 06 19:19:23.099192 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.099150 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:19:23.105100 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.105076 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv"] Jul 06 19:19:23.209904 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.209865 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.210132 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.209930 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80740e3d-ddb4-4c31-9a71-870a1b09acfd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.210132 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.209949 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.210132 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.210000 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.210132 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.210048 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.210132 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.210100 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-678lk\" (UniqueName: \"kubernetes.io/projected/80740e3d-ddb4-4c31-9a71-870a1b09acfd-kube-api-access-678lk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.311495 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.311465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80740e3d-ddb4-4c31-9a71-870a1b09acfd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.311699 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.311503 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.311699 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.311533 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.311699 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.311568 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.311699 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.311599 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-678lk\" (UniqueName: \"kubernetes.io/projected/80740e3d-ddb4-4c31-9a71-870a1b09acfd-kube-api-access-678lk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.311699 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.311636 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.312000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.311975 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.312056 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.311996 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.312108 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.312064 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.313978 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.313955 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80740e3d-ddb4-4c31-9a71-870a1b09acfd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.314119 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.314102 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80740e3d-ddb4-4c31-9a71-870a1b09acfd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.320099 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.320076 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-678lk\" (UniqueName: \"kubernetes.io/projected/80740e3d-ddb4-4c31-9a71-870a1b09acfd-kube-api-access-678lk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w7xv\" (UID: \"80740e3d-ddb4-4c31-9a71-870a1b09acfd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.406150 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.406062 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:23.536986 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.536918 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv"] Jul 06 19:19:23.539449 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:19:23.539420 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80740e3d_ddb4_4c31_9a71_870a1b09acfd.slice/crio-f31833ec921410bf9f5252f342c31f98b752278d923874fc36490903f5588d02 WatchSource:0}: Error finding container f31833ec921410bf9f5252f342c31f98b752278d923874fc36490903f5588d02: Status 404 returned error can't find the container with id f31833ec921410bf9f5252f342c31f98b752278d923874fc36490903f5588d02 Jul 06 19:19:23.894587 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:23.894552 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:19:24.367228 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:24.367185 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" event={"ID":"80740e3d-ddb4-4c31-9a71-870a1b09acfd","Type":"ContainerStarted","Data":"f31833ec921410bf9f5252f342c31f98b752278d923874fc36490903f5588d02"} Jul 06 19:19:25.372904 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:25.372867 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" event={"ID":"80740e3d-ddb4-4c31-9a71-870a1b09acfd","Type":"ContainerStarted","Data":"14ee42d95c4e2a62b030f98fac8950a535d62b768ade030eca13da48a416d5bb"} Jul 06 19:19:25.390267 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:25.390210 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" podStartSLOduration=1.401097004 podStartE2EDuration="2.39019028s" totalImagesPullingTime="989.093276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:19:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:19:25.389236733 +0000 UTC m=+590.872460307" watchObservedRunningTime="2026-07-06 19:19:25.39019028 +0000 UTC m=+590.873413853" Jul 06 19:19:33.406412 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:33.406371 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:33.406869 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:33.406465 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:33.419333 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:33.419303 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:34.425453 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:34.425424 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w7xv" Jul 06 19:19:36.405470 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:36.405441 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:19:36.412025 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:36.411998 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:19:37.993213 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:37.993174 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:19:50.803384 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:19:50.803340 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:20:08.792473 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:20:08.792437 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:20:36.090293 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:20:36.090251 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:21:24.953504 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:24.953468 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f97c9fc9c-d96hp"] Jul 06 19:21:24.956004 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:24.955982 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" Jul 06 19:21:24.959969 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:24.959947 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 19:21:24.959969 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:24.959967 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7q8l6\"" Jul 06 19:21:24.965954 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:24.965932 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f97c9fc9c-d96hp"] Jul 06 19:21:25.010873 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.010835 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84d4deb3-8dea-4149-89f9-81408f9da6fd-tls-cert\") pod \"authorino-7f97c9fc9c-d96hp\" (UID: \"84d4deb3-8dea-4149-89f9-81408f9da6fd\") " pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" Jul 06 19:21:25.011052 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.010886 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwsx\" (UniqueName: \"kubernetes.io/projected/84d4deb3-8dea-4149-89f9-81408f9da6fd-kube-api-access-dpwsx\") pod \"authorino-7f97c9fc9c-d96hp\" (UID: \"84d4deb3-8dea-4149-89f9-81408f9da6fd\") " pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" Jul 06 19:21:25.111844 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.111796 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwsx\" (UniqueName: \"kubernetes.io/projected/84d4deb3-8dea-4149-89f9-81408f9da6fd-kube-api-access-dpwsx\") pod \"authorino-7f97c9fc9c-d96hp\" (UID: \"84d4deb3-8dea-4149-89f9-81408f9da6fd\") " pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" Jul 06 19:21:25.112034 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.111939 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84d4deb3-8dea-4149-89f9-81408f9da6fd-tls-cert\") pod \"authorino-7f97c9fc9c-d96hp\" (UID: \"84d4deb3-8dea-4149-89f9-81408f9da6fd\") " pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" Jul 06 19:21:25.114393 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.114362 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/84d4deb3-8dea-4149-89f9-81408f9da6fd-tls-cert\") pod \"authorino-7f97c9fc9c-d96hp\" (UID: \"84d4deb3-8dea-4149-89f9-81408f9da6fd\") " pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" Jul 06 19:21:25.122652 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.122629 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwsx\" (UniqueName: \"kubernetes.io/projected/84d4deb3-8dea-4149-89f9-81408f9da6fd-kube-api-access-dpwsx\") pod \"authorino-7f97c9fc9c-d96hp\" (UID: \"84d4deb3-8dea-4149-89f9-81408f9da6fd\") " pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" Jul 06 19:21:25.264942 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.264850 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" Jul 06 19:21:25.406115 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.406080 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f97c9fc9c-d96hp"] Jul 06 19:21:25.409331 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:21:25.409305 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84d4deb3_8dea_4149_89f9_81408f9da6fd.slice/crio-2fd270f233385ed83b5f5e4c99984ae20416a630e497c68427a9f526f7224fc0 WatchSource:0}: Error finding container 2fd270f233385ed83b5f5e4c99984ae20416a630e497c68427a9f526f7224fc0: Status 404 returned error can't find the container with id 2fd270f233385ed83b5f5e4c99984ae20416a630e497c68427a9f526f7224fc0 Jul 06 19:21:25.410657 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.410640 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:21:25.920998 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.920963 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" event={"ID":"84d4deb3-8dea-4149-89f9-81408f9da6fd","Type":"ContainerStarted","Data":"82f4cb22c283f450153f91d7af18c9756222c48855025eb0b8aee3fe83d4064e"} Jul 06 19:21:25.921204 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.921008 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" event={"ID":"84d4deb3-8dea-4149-89f9-81408f9da6fd","Type":"ContainerStarted","Data":"2fd270f233385ed83b5f5e4c99984ae20416a630e497c68427a9f526f7224fc0"} Jul 06 19:21:25.940549 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:25.940498 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f97c9fc9c-d96hp" podStartSLOduration=1.598306124 podStartE2EDuration="1.940480313s" totalImagesPullingTime="342.174189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:21:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:21:25.940279398 +0000 UTC m=+711.423502988" watchObservedRunningTime="2026-07-06 19:21:25.940480313 +0000 UTC m=+711.423703891" Jul 06 19:21:53.418295 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:21:53.418212 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:22:04.503879 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:22:04.503841 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:22:15.821272 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:22:15.821236 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:22:24.429914 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:22:24.429879 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:22:35.503744 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:22:35.503705 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:23:02.897647 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:23:02.897602 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:23:14.497616 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:23:14.497524 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:23:49.207440 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:23:49.207404 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:23:56.307625 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:23:56.307587 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:24:04.710374 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:24:04.710333 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:24:11.597496 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:24:11.597458 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:24:23.501703 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:24:23.501659 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:24:34.498275 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:24:34.498236 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:24:36.435989 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:24:36.435959 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:24:36.440438 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:24:36.440418 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:24:46.001787 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:24:46.001749 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:24:53.105679 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:24:53.105641 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:25:00.006917 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:25:00.006878 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:25:09.297320 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:25:09.297282 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:25:21.708012 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:25:21.707976 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:25:30.496791 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:25:30.496753 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:25:56.899001 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:25:56.898921 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:26:06.804629 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:26:06.804591 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:26:21.211449 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:26:21.211409 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:26:31.044006 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:26:31.043969 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:26:42.746677 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:26:42.746632 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:26:51.602792 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:26:51.602757 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:27:00.210172 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:27:00.206484 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:28:49.038383 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:28:49.038346 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:28:58.906022 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:28:58.905982 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:29:09.203093 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:29:09.203050 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:29:21.445775 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:29:21.445720 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:29:31.901492 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:29:31.901458 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:29:36.468056 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:29:36.468032 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:29:36.470041 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:29:36.470019 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:29:41.606364 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:29:41.606335 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:29:50.996936 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:29:50.996903 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:30:00.166410 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:00.166375 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722770-gzxl4"] Jul 06 19:30:00.169796 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:00.169779 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722770-gzxl4" Jul 06 19:30:00.175579 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:00.175560 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-kjghq\"" Jul 06 19:30:00.185142 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:00.185119 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722770-gzxl4"] Jul 06 19:30:00.344499 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:00.344460 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqns\" (UniqueName: \"kubernetes.io/projected/4a7d4507-b124-4813-8a96-6e86b6b0a4f8-kube-api-access-fpqns\") pod \"maas-api-key-cleanup-29722770-gzxl4\" (UID: \"4a7d4507-b124-4813-8a96-6e86b6b0a4f8\") " pod="opendatahub/maas-api-key-cleanup-29722770-gzxl4" Jul 06 19:30:00.445791 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:00.445684 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqns\" (UniqueName: \"kubernetes.io/projected/4a7d4507-b124-4813-8a96-6e86b6b0a4f8-kube-api-access-fpqns\") pod \"maas-api-key-cleanup-29722770-gzxl4\" (UID: \"4a7d4507-b124-4813-8a96-6e86b6b0a4f8\") " pod="opendatahub/maas-api-key-cleanup-29722770-gzxl4" Jul 06 19:30:00.455556 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:00.455531 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqns\" (UniqueName: \"kubernetes.io/projected/4a7d4507-b124-4813-8a96-6e86b6b0a4f8-kube-api-access-fpqns\") pod \"maas-api-key-cleanup-29722770-gzxl4\" (UID: \"4a7d4507-b124-4813-8a96-6e86b6b0a4f8\") " pod="opendatahub/maas-api-key-cleanup-29722770-gzxl4" Jul 06 19:30:00.479213 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:00.479189 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722770-gzxl4" Jul 06 19:30:00.615110 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:00.615085 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722770-gzxl4"] Jul 06 19:30:00.617410 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:30:00.617387 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7d4507_b124_4813_8a96_6e86b6b0a4f8.slice/crio-718df8599ac8fd28587e7e497ba4e63ebf439b77cb06e929d3e4f9d5891d1b51 WatchSource:0}: Error finding container 718df8599ac8fd28587e7e497ba4e63ebf439b77cb06e929d3e4f9d5891d1b51: Status 404 returned error can't find the container with id 718df8599ac8fd28587e7e497ba4e63ebf439b77cb06e929d3e4f9d5891d1b51 Jul 06 19:30:00.619258 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:00.619240 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:30:01.301613 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:01.301577 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722770-gzxl4" event={"ID":"4a7d4507-b124-4813-8a96-6e86b6b0a4f8","Type":"ContainerStarted","Data":"718df8599ac8fd28587e7e497ba4e63ebf439b77cb06e929d3e4f9d5891d1b51"} Jul 06 19:30:01.609549 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:01.609514 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:30:04.318211 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:04.318178 2554 generic.go:356] "Generic (PLEG): container finished" podID="4a7d4507-b124-4813-8a96-6e86b6b0a4f8" containerID="578e82381416ff15c14ea1710496b9ccdfec4da0c112c96958506ea8d5b00911" exitCode=0 Jul 06 19:30:04.318574 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:04.318259 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722770-gzxl4" event={"ID":"4a7d4507-b124-4813-8a96-6e86b6b0a4f8","Type":"ContainerDied","Data":"578e82381416ff15c14ea1710496b9ccdfec4da0c112c96958506ea8d5b00911"} Jul 06 19:30:05.455553 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:05.455530 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722770-gzxl4" Jul 06 19:30:05.489683 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:05.489651 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpqns\" (UniqueName: \"kubernetes.io/projected/4a7d4507-b124-4813-8a96-6e86b6b0a4f8-kube-api-access-fpqns\") pod \"4a7d4507-b124-4813-8a96-6e86b6b0a4f8\" (UID: \"4a7d4507-b124-4813-8a96-6e86b6b0a4f8\") " Jul 06 19:30:05.491791 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:05.491763 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7d4507-b124-4813-8a96-6e86b6b0a4f8-kube-api-access-fpqns" (OuterVolumeSpecName: "kube-api-access-fpqns") pod "4a7d4507-b124-4813-8a96-6e86b6b0a4f8" (UID: "4a7d4507-b124-4813-8a96-6e86b6b0a4f8"). InnerVolumeSpecName "kube-api-access-fpqns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:30:05.590575 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:05.590494 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpqns\" (UniqueName: \"kubernetes.io/projected/4a7d4507-b124-4813-8a96-6e86b6b0a4f8-kube-api-access-fpqns\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:30:06.328715 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:06.328679 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722770-gzxl4" event={"ID":"4a7d4507-b124-4813-8a96-6e86b6b0a4f8","Type":"ContainerDied","Data":"718df8599ac8fd28587e7e497ba4e63ebf439b77cb06e929d3e4f9d5891d1b51"} Jul 06 19:30:06.328715 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:06.328702 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722770-gzxl4" Jul 06 19:30:06.328942 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:06.328751 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="718df8599ac8fd28587e7e497ba4e63ebf439b77cb06e929d3e4f9d5891d1b51" Jul 06 19:30:12.700781 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:12.700750 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:30:21.908412 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:21.908377 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:30:31.703260 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:31.703174 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:30:54.703612 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:30:54.703575 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:31:03.523838 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:31:03.523723 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:31:13.596899 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:31:13.596860 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:31:25.301718 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:31:25.301675 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:31:40.105561 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:31:40.105519 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:31:48.819527 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:31:48.819491 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:31:57.506822 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:31:57.506712 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:32:07.106176 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:32:07.106136 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:32:15.807522 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:32:15.807483 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:32:25.905857 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:32:25.905815 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:32:39.111611 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:32:39.111567 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:32:48.111426 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:32:48.111391 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:32:58.612519 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:32:58.612484 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:33:10.499217 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:33:10.499182 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:33:21.103423 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:33:21.103380 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:33:28.601396 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:33:28.601314 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:33:39.401408 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:33:39.401365 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:33:52.028076 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:33:52.028044 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:34:04.905195 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:34:04.905159 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:34:17.110614 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:34:17.110572 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:34:25.753214 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:34:25.753180 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:34:36.009537 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:34:36.009497 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:34:36.497777 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:34:36.497746 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:34:36.497954 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:34:36.497782 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:34:53.211699 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:34:53.211668 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:35:04.393810 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:35:04.393777 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:35:11.799651 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:35:11.799609 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:35:23.303322 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:35:23.303284 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:36:42.005229 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:36:42.005147 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:36:49.413180 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:36:49.413141 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:36:58.331179 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:36:58.331144 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:37:08.705960 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:08.705920 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:37:16.912511 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:16.912474 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:37:49.744569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:49.744530 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:37:57.499933 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.499857 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g"] Jul 06 19:37:57.500351 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.500212 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a7d4507-b124-4813-8a96-6e86b6b0a4f8" containerName="cleanup" Jul 06 19:37:57.500351 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.500223 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a7d4507-b124-4813-8a96-6e86b6b0a4f8" containerName="cleanup" Jul 06 19:37:57.500351 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.500299 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a7d4507-b124-4813-8a96-6e86b6b0a4f8" containerName="cleanup" Jul 06 19:37:57.503394 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.503370 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.507754 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.507717 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7f0c7a26\"/\"openshift-service-ca.crt\"" Jul 06 19:37:57.507905 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.507806 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7f0c7a26\"/\"default-dockercfg-lrf22\"" Jul 06 19:37:57.508034 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.507899 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7f0c7a26\"/\"e2e-lifecycle-model-7f0c7a26-kserve-self-signed-certs\"" Jul 06 19:37:57.508118 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.508055 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7f0c7a26\"/\"kube-root-ca.crt\"" Jul 06 19:37:57.517269 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.517242 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g"] Jul 06 19:37:57.561920 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.561887 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tls-certs\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.561920 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.561922 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkgg\" (UniqueName: \"kubernetes.io/projected/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-kube-api-access-6gkgg\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.562128 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.561944 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tmp-dir\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.562128 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.561974 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-dshm\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.562128 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.562017 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-model-cache\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.562128 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.562064 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-home\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.663383 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.663346 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-model-cache\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.663561 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.663406 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-home\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.663561 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.663445 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tls-certs\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.663561 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.663462 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkgg\" (UniqueName: \"kubernetes.io/projected/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-kube-api-access-6gkgg\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.663561 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.663485 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tmp-dir\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.663561 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.663518 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-dshm\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.663869 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.663844 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-model-cache\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.663931 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.663875 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-home\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.663931 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.663892 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tmp-dir\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.665708 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.665686 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-dshm\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.665889 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.665874 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tls-certs\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.672273 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.672253 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkgg\" (UniqueName: \"kubernetes.io/projected/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-kube-api-access-6gkgg\") pod \"e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.815324 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.815291 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:37:57.950620 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:57.950596 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g"] Jul 06 19:37:57.952462 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:37:57.952436 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac374dd3_f697_4e7e_9b24_8bb3bdc59e6e.slice/crio-1599b6b1b0a5ccd2c88f289ef4a99da7ed412e52037d4e6854aab6321be2d158 WatchSource:0}: Error finding container 1599b6b1b0a5ccd2c88f289ef4a99da7ed412e52037d4e6854aab6321be2d158: Status 404 returned error can't find the container with id 1599b6b1b0a5ccd2c88f289ef4a99da7ed412e52037d4e6854aab6321be2d158 Jul 06 19:37:58.551819 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:58.551777 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" event={"ID":"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e","Type":"ContainerStarted","Data":"e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b"} Jul 06 19:37:58.552200 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:58.551828 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" event={"ID":"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e","Type":"ContainerStarted","Data":"1599b6b1b0a5ccd2c88f289ef4a99da7ed412e52037d4e6854aab6321be2d158"} Jul 06 19:37:58.574695 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:37:58.574643 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" podStartSLOduration=1.574624294 podStartE2EDuration="1.574624294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:58.572942326 +0000 UTC m=+1704.056165899" watchObservedRunningTime="2026-07-06 19:37:58.574624294 +0000 UTC m=+1704.057847845" Jul 06 19:38:03.204659 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:03.204604 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:38:07.815956 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:07.815903 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:38:07.816380 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:07.815976 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:38:07.828381 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:07.828356 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:38:08.611108 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:08.611079 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:38:17.700540 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:17.700503 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:38:30.907549 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:30.907513 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:38:31.164451 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:31.164353 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g"] Jul 06 19:38:46.184335 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.184284 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" podUID="ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" containerName="main" containerID="cri-o://e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b" gracePeriod=45 Jul 06 19:38:46.429165 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.429141 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:38:46.490773 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.490652 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gkgg\" (UniqueName: \"kubernetes.io/projected/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-kube-api-access-6gkgg\") pod \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " Jul 06 19:38:46.490773 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.490698 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-home\") pod \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " Jul 06 19:38:46.490993 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.490778 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tls-certs\") pod \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " Jul 06 19:38:46.490993 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.490797 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tmp-dir\") pod \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " Jul 06 19:38:46.490993 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.490813 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-dshm\") pod \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " Jul 06 19:38:46.490993 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.490841 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-model-cache\") pod \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\" (UID: \"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e\") " Jul 06 19:38:46.491203 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.491049 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-home" (OuterVolumeSpecName: "home") pod "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" (UID: "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:38:46.491203 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.491156 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:38:46.491300 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.491237 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" (UID: "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:38:46.491300 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.491285 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" (UID: "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:38:46.493048 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.493025 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-dshm" (OuterVolumeSpecName: "dshm") pod "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" (UID: "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:38:46.493212 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.493192 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" (UID: "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:38:46.493265 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.493197 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-kube-api-access-6gkgg" (OuterVolumeSpecName: "kube-api-access-6gkgg") pod "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" (UID: "ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e"). InnerVolumeSpecName "kube-api-access-6gkgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:38:46.592109 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.592061 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:38:46.592109 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.592105 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gkgg\" (UniqueName: \"kubernetes.io/projected/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-kube-api-access-6gkgg\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:38:46.592109 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.592114 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:38:46.592333 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.592122 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:38:46.592333 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.592130 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:38:46.775909 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.775818 2554 generic.go:356] "Generic (PLEG): container finished" podID="ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" containerID="e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b" exitCode=0 Jul 06 19:38:46.775909 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.775868 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" Jul 06 19:38:46.775909 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.775890 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" event={"ID":"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e","Type":"ContainerDied","Data":"e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b"} Jul 06 19:38:46.776166 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.775933 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g" event={"ID":"ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e","Type":"ContainerDied","Data":"1599b6b1b0a5ccd2c88f289ef4a99da7ed412e52037d4e6854aab6321be2d158"} Jul 06 19:38:46.776166 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.775950 2554 scope.go:117] "RemoveContainer" containerID="e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b" Jul 06 19:38:46.784425 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.784400 2554 scope.go:117] "RemoveContainer" containerID="e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b" Jul 06 19:38:46.784687 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:38:46.784664 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b\": container with ID starting with e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b not found: ID does not exist" containerID="e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b" Jul 06 19:38:46.784768 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.784696 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b"} err="failed to get container status \"e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b\": rpc error: code = NotFound desc = could not find container \"e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b\": container with ID starting with e8d3d3c0b403c5a7638969fd4a4f87b1e8bb73024d2b38e5c5a5820dec58935b not found: ID does not exist" Jul 06 19:38:46.802876 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.802843 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g"] Jul 06 19:38:46.808841 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:46.808816 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7f0c7a26/e2e-lifecycle-model-7f0c7a26-kserve-65c6699bdc-mnv4g"] Jul 06 19:38:48.529866 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:38:48.529833 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" path="/var/lib/kubelet/pods/ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e/volumes" Jul 06 19:39:10.709767 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:39:10.709721 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:39:20.100810 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:39:20.100773 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:39:32.802940 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:39:32.802855 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:39:36.526486 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:39:36.526450 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:39:36.528226 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:39:36.528204 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:40:20.606289 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:20.606249 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:40:34.196879 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.196848 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm"] Jul 06 19:40:34.197321 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.197204 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" containerName="main" Jul 06 19:40:34.197321 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.197214 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" containerName="main" Jul 06 19:40:34.197321 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.197274 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac374dd3-f697-4e7e-9b24-8bb3bdc59e6e" containerName="main" Jul 06 19:40:34.200489 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.200469 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.204783 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.204757 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8cc450\"/\"kube-root-ca.crt\"" Jul 06 19:40:34.204783 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.204776 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8cc450\"/\"openshift-service-ca.crt\"" Jul 06 19:40:34.204964 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.204780 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8cc450\"/\"default-dockercfg-c7fgx\"" Jul 06 19:40:34.204964 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.204802 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8cc450\"/\"test-model-8cc450-kserve-self-signed-certs\"" Jul 06 19:40:34.213560 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.213535 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm"] Jul 06 19:40:34.215630 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.215610 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:40:34.253693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.253666 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-model-cache\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.253840 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.253795 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-home\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.253891 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.253857 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkmdm\" (UniqueName: \"kubernetes.io/projected/5bdcaf33-39e8-4848-9229-2fa89b690222-kube-api-access-lkmdm\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.253935 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.253919 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bdcaf33-39e8-4848-9229-2fa89b690222-tls-certs\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.253985 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.253971 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-tmp-dir\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.254059 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.254045 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-dshm\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.355443 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.355410 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bdcaf33-39e8-4848-9229-2fa89b690222-tls-certs\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.355443 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.355449 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-tmp-dir\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.355688 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.355566 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-dshm\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.355688 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.355643 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-model-cache\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.355688 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.355671 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-home\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.355891 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.355720 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmdm\" (UniqueName: \"kubernetes.io/projected/5bdcaf33-39e8-4848-9229-2fa89b690222-kube-api-access-lkmdm\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.355891 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.355823 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-tmp-dir\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.356049 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.356030 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-model-cache\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.356355 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.356044 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-home\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.357847 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.357827 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-dshm\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.358036 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.358020 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bdcaf33-39e8-4848-9229-2fa89b690222-tls-certs\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.367350 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.367329 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmdm\" (UniqueName: \"kubernetes.io/projected/5bdcaf33-39e8-4848-9229-2fa89b690222-kube-api-access-lkmdm\") pod \"test-model-8cc450-kserve-798bdd94ff-hzknm\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.509989 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.509900 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:34.646795 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:34.646762 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm"] Jul 06 19:40:34.649995 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:40:34.649964 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bdcaf33_39e8_4848_9229_2fa89b690222.slice/crio-0203445a71792f788c6f3a6e575655e2ddc3d52e2f123a0fc5734cdcbcd1f8c1 WatchSource:0}: Error finding container 0203445a71792f788c6f3a6e575655e2ddc3d52e2f123a0fc5734cdcbcd1f8c1: Status 404 returned error can't find the container with id 0203445a71792f788c6f3a6e575655e2ddc3d52e2f123a0fc5734cdcbcd1f8c1 Jul 06 19:40:35.293287 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:35.293249 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" event={"ID":"5bdcaf33-39e8-4848-9229-2fa89b690222","Type":"ContainerStarted","Data":"d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4"} Jul 06 19:40:35.293670 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:35.293295 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" event={"ID":"5bdcaf33-39e8-4848-9229-2fa89b690222","Type":"ContainerStarted","Data":"0203445a71792f788c6f3a6e575655e2ddc3d52e2f123a0fc5734cdcbcd1f8c1"} Jul 06 19:40:35.314858 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:35.314805 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" podStartSLOduration=1.314790443 podStartE2EDuration="1.314790443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:40:35.312836132 +0000 UTC m=+1860.796059697" watchObservedRunningTime="2026-07-06 19:40:35.314790443 +0000 UTC m=+1860.798014015" Jul 06 19:40:44.510569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:44.510519 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:44.510569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:44.510584 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:44.522592 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:44.522566 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:45.353293 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:45.353266 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:40:49.097511 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:40:49.097469 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:41:00.395857 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.395820 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5"] Jul 06 19:41:00.399522 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.399497 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.403950 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.403917 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-930663\"/\"openshift-service-ca.crt\"" Jul 06 19:41:00.404105 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.403947 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-930663\"/\"test-model-930663-kserve-self-signed-certs\"" Jul 06 19:41:00.404105 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.404000 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-930663\"/\"kube-root-ca.crt\"" Jul 06 19:41:00.404230 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.404102 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-930663\"/\"default-dockercfg-q5dz2\"" Jul 06 19:41:00.411744 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.411703 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5"] Jul 06 19:41:00.503305 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.503270 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8psmn\" (UniqueName: \"kubernetes.io/projected/20538cc6-a222-4b29-9f35-ef243632157d-kube-api-access-8psmn\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.503305 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.503309 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-tmp-dir\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.503531 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.503347 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-dshm\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.503531 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.503380 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-home\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.503531 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.503401 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20538cc6-a222-4b29-9f35-ef243632157d-tls-certs\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.503633 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.503528 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-model-cache\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.604764 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.604693 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20538cc6-a222-4b29-9f35-ef243632157d-tls-certs\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.604936 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.604814 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-model-cache\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.604936 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.604870 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8psmn\" (UniqueName: \"kubernetes.io/projected/20538cc6-a222-4b29-9f35-ef243632157d-kube-api-access-8psmn\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.604936 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.604899 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-tmp-dir\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.605121 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.604936 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-dshm\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.605121 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.604971 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-home\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.605344 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.605319 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-tmp-dir\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.605421 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.605352 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-model-cache\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.605421 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.605407 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-home\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.607010 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.606989 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-dshm\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.607292 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.607275 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20538cc6-a222-4b29-9f35-ef243632157d-tls-certs\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.615697 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.615676 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8psmn\" (UniqueName: \"kubernetes.io/projected/20538cc6-a222-4b29-9f35-ef243632157d-kube-api-access-8psmn\") pod \"test-model-930663-kserve-5cf68f58c5-prrp5\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.709661 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.709570 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:00.846144 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:00.846111 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5"] Jul 06 19:41:00.849487 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:41:00.849460 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20538cc6_a222_4b29_9f35_ef243632157d.slice/crio-8ec1986278ce19b6d88cc61e9dc8f7d1c39528715a35672d5648db549e206a44 WatchSource:0}: Error finding container 8ec1986278ce19b6d88cc61e9dc8f7d1c39528715a35672d5648db549e206a44: Status 404 returned error can't find the container with id 8ec1986278ce19b6d88cc61e9dc8f7d1c39528715a35672d5648db549e206a44 Jul 06 19:41:01.429502 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:01.429470 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" event={"ID":"20538cc6-a222-4b29-9f35-ef243632157d","Type":"ContainerStarted","Data":"8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197"} Jul 06 19:41:01.429918 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:01.429511 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" event={"ID":"20538cc6-a222-4b29-9f35-ef243632157d","Type":"ContainerStarted","Data":"8ec1986278ce19b6d88cc61e9dc8f7d1c39528715a35672d5648db549e206a44"} Jul 06 19:41:01.449450 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:01.449397 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" podStartSLOduration=1.4493806870000001 podStartE2EDuration="1.449380687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:41:01.446759652 +0000 UTC m=+1886.929983216" watchObservedRunningTime="2026-07-06 19:41:01.449380687 +0000 UTC m=+1886.932604260" Jul 06 19:41:04.501382 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:04.501339 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:41:10.710146 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:10.710109 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:10.710703 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:10.710173 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:10.727525 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:10.727492 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:11.488484 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:11.488454 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:17.220362 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:17.220328 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:41:24.256038 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:24.256005 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm"] Jul 06 19:41:25.251000 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:25.250965 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5"] Jul 06 19:41:30.497441 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:30.497402 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:41:39.275930 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.275864 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" podUID="5bdcaf33-39e8-4848-9229-2fa89b690222" containerName="main" containerID="cri-o://d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4" gracePeriod=45 Jul 06 19:41:39.518312 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.518290 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:41:39.587304 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587270 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-tmp-dir\") pod \"5bdcaf33-39e8-4848-9229-2fa89b690222\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " Jul 06 19:41:39.587499 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587330 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-home\") pod \"5bdcaf33-39e8-4848-9229-2fa89b690222\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " Jul 06 19:41:39.587499 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587347 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-dshm\") pod \"5bdcaf33-39e8-4848-9229-2fa89b690222\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " Jul 06 19:41:39.587499 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587382 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-model-cache\") pod \"5bdcaf33-39e8-4848-9229-2fa89b690222\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " Jul 06 19:41:39.587499 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587432 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bdcaf33-39e8-4848-9229-2fa89b690222-tls-certs\") pod \"5bdcaf33-39e8-4848-9229-2fa89b690222\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " Jul 06 19:41:39.587499 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587464 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkmdm\" (UniqueName: \"kubernetes.io/projected/5bdcaf33-39e8-4848-9229-2fa89b690222-kube-api-access-lkmdm\") pod \"5bdcaf33-39e8-4848-9229-2fa89b690222\" (UID: \"5bdcaf33-39e8-4848-9229-2fa89b690222\") " Jul 06 19:41:39.587792 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587573 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-home" (OuterVolumeSpecName: "home") pod "5bdcaf33-39e8-4848-9229-2fa89b690222" (UID: "5bdcaf33-39e8-4848-9229-2fa89b690222"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:39.587792 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587584 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bdcaf33-39e8-4848-9229-2fa89b690222" (UID: "5bdcaf33-39e8-4848-9229-2fa89b690222"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:39.587792 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587658 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bdcaf33-39e8-4848-9229-2fa89b690222" (UID: "5bdcaf33-39e8-4848-9229-2fa89b690222"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:39.587921 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587805 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:39.587921 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587825 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:39.587921 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.587838 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:39.589673 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.589639 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-dshm" (OuterVolumeSpecName: "dshm") pod "5bdcaf33-39e8-4848-9229-2fa89b690222" (UID: "5bdcaf33-39e8-4848-9229-2fa89b690222"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:39.589673 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.589641 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bdcaf33-39e8-4848-9229-2fa89b690222-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bdcaf33-39e8-4848-9229-2fa89b690222" (UID: "5bdcaf33-39e8-4848-9229-2fa89b690222"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:41:39.589852 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.589693 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdcaf33-39e8-4848-9229-2fa89b690222-kube-api-access-lkmdm" (OuterVolumeSpecName: "kube-api-access-lkmdm") pod "5bdcaf33-39e8-4848-9229-2fa89b690222" (UID: "5bdcaf33-39e8-4848-9229-2fa89b690222"). InnerVolumeSpecName "kube-api-access-lkmdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:41:39.610273 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.610241 2554 generic.go:356] "Generic (PLEG): container finished" podID="5bdcaf33-39e8-4848-9229-2fa89b690222" containerID="d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4" exitCode=0 Jul 06 19:41:39.610384 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.610292 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" Jul 06 19:41:39.610384 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.610325 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" event={"ID":"5bdcaf33-39e8-4848-9229-2fa89b690222","Type":"ContainerDied","Data":"d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4"} Jul 06 19:41:39.610384 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.610369 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm" event={"ID":"5bdcaf33-39e8-4848-9229-2fa89b690222","Type":"ContainerDied","Data":"0203445a71792f788c6f3a6e575655e2ddc3d52e2f123a0fc5734cdcbcd1f8c1"} Jul 06 19:41:39.610486 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.610386 2554 scope.go:117] "RemoveContainer" containerID="d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4" Jul 06 19:41:39.619073 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.619053 2554 scope.go:117] "RemoveContainer" containerID="d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4" Jul 06 19:41:39.619312 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:41:39.619297 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4\": container with ID starting with d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4 not found: ID does not exist" containerID="d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4" Jul 06 19:41:39.619369 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.619319 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4"} err="failed to get container status \"d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4\": rpc error: code = NotFound desc = could not find container \"d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4\": container with ID starting with d306c9d77c4f5284b214e2b140c4b7565de0fbfd6b7b03308b7874e8de08aba4 not found: ID does not exist" Jul 06 19:41:39.636102 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.636068 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm"] Jul 06 19:41:39.640580 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.640554 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8cc450/test-model-8cc450-kserve-798bdd94ff-hzknm"] Jul 06 19:41:39.688653 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.688629 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bdcaf33-39e8-4848-9229-2fa89b690222-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:39.688653 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.688651 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkmdm\" (UniqueName: \"kubernetes.io/projected/5bdcaf33-39e8-4848-9229-2fa89b690222-kube-api-access-lkmdm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:39.688906 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:39.688660 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bdcaf33-39e8-4848-9229-2fa89b690222-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:40.270768 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.270710 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" podUID="20538cc6-a222-4b29-9f35-ef243632157d" containerName="main" containerID="cri-o://8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197" gracePeriod=45 Jul 06 19:41:40.524937 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.524874 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:40.529649 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.529620 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bdcaf33-39e8-4848-9229-2fa89b690222" path="/var/lib/kubelet/pods/5bdcaf33-39e8-4848-9229-2fa89b690222/volumes" Jul 06 19:41:40.596963 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.596930 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-model-cache\") pod \"20538cc6-a222-4b29-9f35-ef243632157d\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " Jul 06 19:41:40.597125 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.597026 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8psmn\" (UniqueName: \"kubernetes.io/projected/20538cc6-a222-4b29-9f35-ef243632157d-kube-api-access-8psmn\") pod \"20538cc6-a222-4b29-9f35-ef243632157d\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " Jul 06 19:41:40.597125 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.597054 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-tmp-dir\") pod \"20538cc6-a222-4b29-9f35-ef243632157d\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " Jul 06 19:41:40.597125 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.597068 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-dshm\") pod \"20538cc6-a222-4b29-9f35-ef243632157d\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " Jul 06 19:41:40.597125 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.597101 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-home\") pod \"20538cc6-a222-4b29-9f35-ef243632157d\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " Jul 06 19:41:40.597337 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.597146 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20538cc6-a222-4b29-9f35-ef243632157d-tls-certs\") pod \"20538cc6-a222-4b29-9f35-ef243632157d\" (UID: \"20538cc6-a222-4b29-9f35-ef243632157d\") " Jul 06 19:41:40.597337 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.597199 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-model-cache" (OuterVolumeSpecName: "model-cache") pod "20538cc6-a222-4b29-9f35-ef243632157d" (UID: "20538cc6-a222-4b29-9f35-ef243632157d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:40.597440 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.597370 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20538cc6-a222-4b29-9f35-ef243632157d" (UID: "20538cc6-a222-4b29-9f35-ef243632157d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:40.597440 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.597380 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-home" (OuterVolumeSpecName: "home") pod "20538cc6-a222-4b29-9f35-ef243632157d" (UID: "20538cc6-a222-4b29-9f35-ef243632157d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:40.597440 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.597432 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:40.599361 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.599332 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20538cc6-a222-4b29-9f35-ef243632157d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20538cc6-a222-4b29-9f35-ef243632157d" (UID: "20538cc6-a222-4b29-9f35-ef243632157d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:41:40.599468 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.599366 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20538cc6-a222-4b29-9f35-ef243632157d-kube-api-access-8psmn" (OuterVolumeSpecName: "kube-api-access-8psmn") pod "20538cc6-a222-4b29-9f35-ef243632157d" (UID: "20538cc6-a222-4b29-9f35-ef243632157d"). InnerVolumeSpecName "kube-api-access-8psmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:41:40.599468 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.599396 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-dshm" (OuterVolumeSpecName: "dshm") pod "20538cc6-a222-4b29-9f35-ef243632157d" (UID: "20538cc6-a222-4b29-9f35-ef243632157d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:41:40.616427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.616405 2554 generic.go:356] "Generic (PLEG): container finished" podID="20538cc6-a222-4b29-9f35-ef243632157d" containerID="8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197" exitCode=0 Jul 06 19:41:40.616532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.616446 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" Jul 06 19:41:40.616532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.616478 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" event={"ID":"20538cc6-a222-4b29-9f35-ef243632157d","Type":"ContainerDied","Data":"8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197"} Jul 06 19:41:40.616532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.616512 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5" event={"ID":"20538cc6-a222-4b29-9f35-ef243632157d","Type":"ContainerDied","Data":"8ec1986278ce19b6d88cc61e9dc8f7d1c39528715a35672d5648db549e206a44"} Jul 06 19:41:40.616532 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.616529 2554 scope.go:117] "RemoveContainer" containerID="8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197" Jul 06 19:41:40.625385 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.625366 2554 scope.go:117] "RemoveContainer" containerID="8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197" Jul 06 19:41:40.625647 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:41:40.625630 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197\": container with ID starting with 8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197 not found: ID does not exist" containerID="8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197" Jul 06 19:41:40.625715 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.625658 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197"} err="failed to get container status \"8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197\": rpc error: code = NotFound desc = could not find container \"8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197\": container with ID starting with 8c4be454fa9d8d09e17d3802cf2509608dcd8e0f0108f0f487bab4cf9dd4b197 not found: ID does not exist" Jul 06 19:41:40.650663 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.650635 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5"] Jul 06 19:41:40.657184 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.657164 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-930663/test-model-930663-kserve-5cf68f58c5-prrp5"] Jul 06 19:41:40.698026 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.698007 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:40.698026 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.698027 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20538cc6-a222-4b29-9f35-ef243632157d-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:40.698165 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.698037 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8psmn\" (UniqueName: \"kubernetes.io/projected/20538cc6-a222-4b29-9f35-ef243632157d-kube-api-access-8psmn\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:40.698165 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.698046 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:40.698165 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:40.698057 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20538cc6-a222-4b29-9f35-ef243632157d-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:41:42.530276 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:41:42.530242 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20538cc6-a222-4b29-9f35-ef243632157d" path="/var/lib/kubelet/pods/20538cc6-a222-4b29-9f35-ef243632157d/volumes" Jul 06 19:42:12.110414 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:42:12.110379 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:42:26.011939 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:42:26.011890 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:42:43.702540 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:42:43.702506 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:42:57.004417 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:42:57.004382 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:43:09.405899 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:09.405867 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:43:21.797248 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.797215 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z"] Jul 06 19:43:21.797719 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.797558 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20538cc6-a222-4b29-9f35-ef243632157d" containerName="main" Jul 06 19:43:21.797719 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.797569 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="20538cc6-a222-4b29-9f35-ef243632157d" containerName="main" Jul 06 19:43:21.797719 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.797586 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bdcaf33-39e8-4848-9229-2fa89b690222" containerName="main" Jul 06 19:43:21.797719 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.797591 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bdcaf33-39e8-4848-9229-2fa89b690222" containerName="main" Jul 06 19:43:21.797719 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.797651 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="20538cc6-a222-4b29-9f35-ef243632157d" containerName="main" Jul 06 19:43:21.797719 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.797664 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bdcaf33-39e8-4848-9229-2fa89b690222" containerName="main" Jul 06 19:43:21.800791 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.800773 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.803791 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.803772 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-26761c\"/\"default-dockercfg-t4rpq\"" Jul 06 19:43:21.804942 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.804926 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-26761c\"/\"auth-test-model-26761c-kserve-self-signed-certs\"" Jul 06 19:43:21.805016 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.804969 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-26761c\"/\"kube-root-ca.crt\"" Jul 06 19:43:21.805016 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.804928 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-26761c\"/\"openshift-service-ca.crt\"" Jul 06 19:43:21.817708 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.817683 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z"] Jul 06 19:43:21.883768 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.883715 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-dshm\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.883960 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.883827 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-home\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.883960 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.883862 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tmp-dir\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.883960 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.883890 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tls-certs\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.883960 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.883950 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-model-cache\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.884188 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.884015 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqb2m\" (UniqueName: \"kubernetes.io/projected/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-kube-api-access-cqb2m\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.984646 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.984612 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-home\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.984646 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.984648 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tmp-dir\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.984921 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.984666 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tls-certs\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.984921 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.984690 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-model-cache\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.984921 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.984747 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqb2m\" (UniqueName: \"kubernetes.io/projected/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-kube-api-access-cqb2m\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.984921 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.984791 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-dshm\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.985156 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.985005 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-home\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.985156 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.985096 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tmp-dir\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.985156 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.985127 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-model-cache\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.986977 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.986949 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-dshm\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.987277 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.987257 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tls-certs\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:21.993913 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:21.993889 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqb2m\" (UniqueName: \"kubernetes.io/projected/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-kube-api-access-cqb2m\") pod \"auth-test-model-26761c-kserve-75574d5595-dlb5z\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:22.110271 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:22.110242 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:22.267362 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:22.267335 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z"] Jul 06 19:43:22.268560 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:43:22.268536 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e9f3397_fe0d_4f1b_91b8_f099667f98eb.slice/crio-38e3b9ef8705ba848e1785ab5cd8e59282fed202709ae933149fbc5f533f1dec WatchSource:0}: Error finding container 38e3b9ef8705ba848e1785ab5cd8e59282fed202709ae933149fbc5f533f1dec: Status 404 returned error can't find the container with id 38e3b9ef8705ba848e1785ab5cd8e59282fed202709ae933149fbc5f533f1dec Jul 06 19:43:23.095132 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:23.095093 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" event={"ID":"7e9f3397-fe0d-4f1b-91b8-f099667f98eb","Type":"ContainerStarted","Data":"f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21"} Jul 06 19:43:23.095132 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:23.095136 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" event={"ID":"7e9f3397-fe0d-4f1b-91b8-f099667f98eb","Type":"ContainerStarted","Data":"38e3b9ef8705ba848e1785ab5cd8e59282fed202709ae933149fbc5f533f1dec"} Jul 06 19:43:23.129829 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:23.129780 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" podStartSLOduration=2.129764337 podStartE2EDuration="2.129764337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:43:23.128896345 +0000 UTC m=+2028.612119917" watchObservedRunningTime="2026-07-06 19:43:23.129764337 +0000 UTC m=+2028.612987904" Jul 06 19:43:26.918550 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:26.918511 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:43:32.110522 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:32.110483 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:32.111066 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:32.110682 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:32.122956 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:32.122933 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:43:32.146123 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:43:32.146096 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:44:01.397427 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.397388 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv"] Jul 06 19:44:01.401255 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.401232 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.404402 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.404379 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"default-dockercfg-rvh7f\"" Jul 06 19:44:01.404501 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.404382 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"openshift-service-ca.crt\"" Jul 06 19:44:01.405888 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.405868 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"kube-root-ca.crt\"" Jul 06 19:44:01.405888 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.405884 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"auth-test-model-bb8c15-kserve-self-signed-certs\"" Jul 06 19:44:01.413262 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.413237 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv"] Jul 06 19:44:01.549685 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.549651 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tmp-dir\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.549893 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.549692 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-dshm\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.549893 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.549828 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-model-cache\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.549893 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.549855 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tls-certs\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.549893 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.549876 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dcqd\" (UniqueName: \"kubernetes.io/projected/b7236594-41e5-4fbe-babb-73c1ce00cdf1-kube-api-access-8dcqd\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.550033 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.549948 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-home\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.650478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.650401 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tmp-dir\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.650478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.650436 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-dshm\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.650478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.650477 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-model-cache\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.650693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.650496 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tls-certs\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.650693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.650513 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dcqd\" (UniqueName: \"kubernetes.io/projected/b7236594-41e5-4fbe-babb-73c1ce00cdf1-kube-api-access-8dcqd\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.650693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.650599 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-home\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.650924 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.650889 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tmp-dir\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.651007 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.650919 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-model-cache\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.651007 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.650983 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-home\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.652893 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.652869 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-dshm\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.653094 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.653080 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tls-certs\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.658839 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.658818 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dcqd\" (UniqueName: \"kubernetes.io/projected/b7236594-41e5-4fbe-babb-73c1ce00cdf1-kube-api-access-8dcqd\") pod \"auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.711681 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.711644 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:01.845992 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:01.845969 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv"] Jul 06 19:44:01.848156 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:44:01.848123 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7236594_41e5_4fbe_babb_73c1ce00cdf1.slice/crio-6806cff13b9448ed9805705eaadc4a5e95f39591a4ca6031a18557264934a622 WatchSource:0}: Error finding container 6806cff13b9448ed9805705eaadc4a5e95f39591a4ca6031a18557264934a622: Status 404 returned error can't find the container with id 6806cff13b9448ed9805705eaadc4a5e95f39591a4ca6031a18557264934a622 Jul 06 19:44:02.276327 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:02.276288 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" event={"ID":"b7236594-41e5-4fbe-babb-73c1ce00cdf1","Type":"ContainerStarted","Data":"375503048b4717f1d54e7d2ccc64312453bf89c2e44c9968f4d55456f11f13c7"} Jul 06 19:44:02.276482 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:02.276333 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" event={"ID":"b7236594-41e5-4fbe-babb-73c1ce00cdf1","Type":"ContainerStarted","Data":"6806cff13b9448ed9805705eaadc4a5e95f39591a4ca6031a18557264934a622"} Jul 06 19:44:02.306914 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:02.306849 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" podStartSLOduration=1.306833272 podStartE2EDuration="1.306833272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:44:02.306617185 +0000 UTC m=+2067.789840760" watchObservedRunningTime="2026-07-06 19:44:02.306833272 +0000 UTC m=+2067.790056844" Jul 06 19:44:04.506569 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:04.506535 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:44:11.712676 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:11.712638 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:11.713169 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:11.712692 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:11.724951 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:11.724928 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:12.332357 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:12.332326 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:44:23.121294 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:23.121260 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:44:36.561116 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:36.561091 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:44:36.563800 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:36.563780 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:44:38.502913 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:38.502872 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:44:55.607800 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:44:55.607761 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:45:09.813081 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:45:09.813032 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:45:26.503761 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:45:26.503661 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:45:42.709716 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:45:42.709677 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:45:56.907668 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:45:56.907635 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z"] Jul 06 19:45:57.689025 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:45:57.688988 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv"] Jul 06 19:45:59.613996 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:45:59.613957 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:46:11.927102 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:11.927040 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" podUID="7e9f3397-fe0d-4f1b-91b8-f099667f98eb" containerName="main" containerID="cri-o://f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21" gracePeriod=45 Jul 06 19:46:12.176925 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.176892 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:46:12.361693 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.361663 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-home\") pod \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " Jul 06 19:46:12.361861 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.361806 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-model-cache\") pod \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " Jul 06 19:46:12.361861 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.361849 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tmp-dir\") pod \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " Jul 06 19:46:12.361982 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.361892 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tls-certs\") pod \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " Jul 06 19:46:12.361982 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.361915 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqb2m\" (UniqueName: \"kubernetes.io/projected/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-kube-api-access-cqb2m\") pod \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " Jul 06 19:46:12.361982 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.361917 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-home" (OuterVolumeSpecName: "home") pod "7e9f3397-fe0d-4f1b-91b8-f099667f98eb" (UID: "7e9f3397-fe0d-4f1b-91b8-f099667f98eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:12.361982 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.361940 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-dshm\") pod \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\" (UID: \"7e9f3397-fe0d-4f1b-91b8-f099667f98eb\") " Jul 06 19:46:12.362182 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.362065 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e9f3397-fe0d-4f1b-91b8-f099667f98eb" (UID: "7e9f3397-fe0d-4f1b-91b8-f099667f98eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:12.362182 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.362159 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e9f3397-fe0d-4f1b-91b8-f099667f98eb" (UID: "7e9f3397-fe0d-4f1b-91b8-f099667f98eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:12.362292 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.362270 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:12.362292 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.362286 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:12.362376 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.362298 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:12.364178 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.364155 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-kube-api-access-cqb2m" (OuterVolumeSpecName: "kube-api-access-cqb2m") pod "7e9f3397-fe0d-4f1b-91b8-f099667f98eb" (UID: "7e9f3397-fe0d-4f1b-91b8-f099667f98eb"). InnerVolumeSpecName "kube-api-access-cqb2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:46:12.364446 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.364424 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-dshm" (OuterVolumeSpecName: "dshm") pod "7e9f3397-fe0d-4f1b-91b8-f099667f98eb" (UID: "7e9f3397-fe0d-4f1b-91b8-f099667f98eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:12.364526 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.364439 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e9f3397-fe0d-4f1b-91b8-f099667f98eb" (UID: "7e9f3397-fe0d-4f1b-91b8-f099667f98eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:46:12.463137 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.463100 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:12.463137 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.463130 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqb2m\" (UniqueName: \"kubernetes.io/projected/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-kube-api-access-cqb2m\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:12.463137 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.463139 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9f3397-fe0d-4f1b-91b8-f099667f98eb-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:12.708464 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.708376 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" podUID="b7236594-41e5-4fbe-babb-73c1ce00cdf1" containerName="main" containerID="cri-o://375503048b4717f1d54e7d2ccc64312453bf89c2e44c9968f4d55456f11f13c7" gracePeriod=45 Jul 06 19:46:12.873224 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.873188 2554 generic.go:356] "Generic (PLEG): container finished" podID="7e9f3397-fe0d-4f1b-91b8-f099667f98eb" containerID="f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21" exitCode=0 Jul 06 19:46:12.873373 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.873224 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" event={"ID":"7e9f3397-fe0d-4f1b-91b8-f099667f98eb","Type":"ContainerDied","Data":"f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21"} Jul 06 19:46:12.873373 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.873283 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" event={"ID":"7e9f3397-fe0d-4f1b-91b8-f099667f98eb","Type":"ContainerDied","Data":"38e3b9ef8705ba848e1785ab5cd8e59282fed202709ae933149fbc5f533f1dec"} Jul 06 19:46:12.873373 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.873306 2554 scope.go:117] "RemoveContainer" containerID="f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21" Jul 06 19:46:12.873563 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.873527 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" Jul 06 19:46:12.875482 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.875453 2554 generic.go:356] "Generic (PLEG): container finished" podID="b7236594-41e5-4fbe-babb-73c1ce00cdf1" containerID="375503048b4717f1d54e7d2ccc64312453bf89c2e44c9968f4d55456f11f13c7" exitCode=0 Jul 06 19:46:12.875598 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.875519 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" event={"ID":"b7236594-41e5-4fbe-babb-73c1ce00cdf1","Type":"ContainerDied","Data":"375503048b4717f1d54e7d2ccc64312453bf89c2e44c9968f4d55456f11f13c7"} Jul 06 19:46:12.883448 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.883426 2554 scope.go:117] "RemoveContainer" containerID="f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21" Jul 06 19:46:12.883707 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:46:12.883689 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21\": container with ID starting with f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21 not found: ID does not exist" containerID="f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21" Jul 06 19:46:12.883792 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.883716 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21"} err="failed to get container status \"f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21\": rpc error: code = NotFound desc = could not find container \"f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21\": container with ID starting with f3f31468f470db557d06ea22a61b1b768575650cfbc62a11644b73e618c30f21 not found: ID does not exist" Jul 06 19:46:12.893053 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.893028 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z"] Jul 06 19:46:12.899374 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.899357 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z"] Jul 06 19:46:12.957484 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:12.957457 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:46:13.068883 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.068852 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-model-cache\") pod \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " Jul 06 19:46:13.069063 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.068910 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tmp-dir\") pod \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " Jul 06 19:46:13.069063 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.068970 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-dshm\") pod \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " Jul 06 19:46:13.069063 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.068997 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tls-certs\") pod \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " Jul 06 19:46:13.069063 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.069025 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-home\") pod \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " Jul 06 19:46:13.069283 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.069098 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dcqd\" (UniqueName: \"kubernetes.io/projected/b7236594-41e5-4fbe-babb-73c1ce00cdf1-kube-api-access-8dcqd\") pod \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\" (UID: \"b7236594-41e5-4fbe-babb-73c1ce00cdf1\") " Jul 06 19:46:13.069283 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.069128 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7236594-41e5-4fbe-babb-73c1ce00cdf1" (UID: "b7236594-41e5-4fbe-babb-73c1ce00cdf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:13.069283 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.069194 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7236594-41e5-4fbe-babb-73c1ce00cdf1" (UID: "b7236594-41e5-4fbe-babb-73c1ce00cdf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:13.069422 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.069310 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-home" (OuterVolumeSpecName: "home") pod "b7236594-41e5-4fbe-babb-73c1ce00cdf1" (UID: "b7236594-41e5-4fbe-babb-73c1ce00cdf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:13.069422 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.069390 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:13.069422 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.069404 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:13.069422 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.069413 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:13.071327 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.071306 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-dshm" (OuterVolumeSpecName: "dshm") pod "b7236594-41e5-4fbe-babb-73c1ce00cdf1" (UID: "b7236594-41e5-4fbe-babb-73c1ce00cdf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:13.071428 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.071361 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7236594-41e5-4fbe-babb-73c1ce00cdf1" (UID: "b7236594-41e5-4fbe-babb-73c1ce00cdf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:46:13.071465 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.071423 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7236594-41e5-4fbe-babb-73c1ce00cdf1-kube-api-access-8dcqd" (OuterVolumeSpecName: "kube-api-access-8dcqd") pod "b7236594-41e5-4fbe-babb-73c1ce00cdf1" (UID: "b7236594-41e5-4fbe-babb-73c1ce00cdf1"). InnerVolumeSpecName "kube-api-access-8dcqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:46:13.170175 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.170137 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dcqd\" (UniqueName: \"kubernetes.io/projected/b7236594-41e5-4fbe-babb-73c1ce00cdf1-kube-api-access-8dcqd\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:13.170175 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.170168 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7236594-41e5-4fbe-babb-73c1ce00cdf1-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:13.170175 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.170177 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7236594-41e5-4fbe-babb-73c1ce00cdf1-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:46:13.881121 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.881089 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" Jul 06 19:46:13.881339 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.881124 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv" event={"ID":"b7236594-41e5-4fbe-babb-73c1ce00cdf1","Type":"ContainerDied","Data":"6806cff13b9448ed9805705eaadc4a5e95f39591a4ca6031a18557264934a622"} Jul 06 19:46:13.881339 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.881182 2554 scope.go:117] "RemoveContainer" containerID="375503048b4717f1d54e7d2ccc64312453bf89c2e44c9968f4d55456f11f13c7" Jul 06 19:46:13.911032 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.911000 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv"] Jul 06 19:46:13.919027 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:13.919000 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/auth-test-model-bb8c15-kserve-6655f8f9cf-gh2rv"] Jul 06 19:46:14.530018 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:14.529980 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9f3397-fe0d-4f1b-91b8-f099667f98eb" path="/var/lib/kubelet/pods/7e9f3397-fe0d-4f1b-91b8-f099667f98eb/volumes" Jul 06 19:46:14.530477 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:14.530458 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7236594-41e5-4fbe-babb-73c1ce00cdf1" path="/var/lib/kubelet/pods/b7236594-41e5-4fbe-babb-73c1ce00cdf1/volumes" Jul 06 19:46:16.008295 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:16.008262 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:46:17.136252 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:17.136207 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-26761c/auth-test-model-26761c-kserve-75574d5595-dlb5z" podUID="7e9f3397-fe0d-4f1b-91b8-f099667f98eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Jul 06 19:46:37.994540 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:37.994506 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw"] Jul 06 19:46:37.994985 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:37.994900 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9f3397-fe0d-4f1b-91b8-f099667f98eb" containerName="main" Jul 06 19:46:37.994985 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:37.994912 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9f3397-fe0d-4f1b-91b8-f099667f98eb" containerName="main" Jul 06 19:46:37.994985 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:37.994936 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7236594-41e5-4fbe-babb-73c1ce00cdf1" containerName="main" Jul 06 19:46:37.994985 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:37.994942 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7236594-41e5-4fbe-babb-73c1ce00cdf1" containerName="main" Jul 06 19:46:37.995133 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:37.995003 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7236594-41e5-4fbe-babb-73c1ce00cdf1" containerName="main" Jul 06 19:46:37.995133 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:37.995015 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9f3397-fe0d-4f1b-91b8-f099667f98eb" containerName="main" Jul 06 19:46:37.999853 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:37.999832 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.002760 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.002725 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"openshift-service-ca.crt\"" Jul 06 19:46:38.002760 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.002725 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"default-dockercfg-rvh7f\"" Jul 06 19:46:38.004011 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.003996 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"kube-root-ca.crt\"" Jul 06 19:46:38.004151 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.004132 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"sub-test-model-bb8c15-kserve-self-signed-certs\"" Jul 06 19:46:38.008628 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.008604 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw"] Jul 06 19:46:38.094646 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.094606 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-model-cache\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.094646 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.094648 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tmp-dir\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.094915 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.094683 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tls-certs\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.094915 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.094778 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-home\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.094915 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.094808 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-dshm\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.094915 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.094847 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh6gd\" (UniqueName: \"kubernetes.io/projected/730a9db9-28ca-4a2e-a069-fd639a1bdddd-kube-api-access-lh6gd\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.195505 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.195468 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-dshm\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.195668 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.195517 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh6gd\" (UniqueName: \"kubernetes.io/projected/730a9db9-28ca-4a2e-a069-fd639a1bdddd-kube-api-access-lh6gd\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.195668 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.195556 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-model-cache\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.195668 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.195576 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tmp-dir\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.195668 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.195609 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tls-certs\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.195889 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.195755 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-home\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.196004 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.195985 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tmp-dir\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.196081 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.196013 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-model-cache\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.196126 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.196087 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-home\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.197856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.197839 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-dshm\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.198026 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.198009 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tls-certs\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.204790 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.204721 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh6gd\" (UniqueName: \"kubernetes.io/projected/730a9db9-28ca-4a2e-a069-fd639a1bdddd-kube-api-access-lh6gd\") pod \"sub-test-model-bb8c15-kserve-5f84c94765-hvqsw\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.311040 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.310952 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:38.445136 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:38.445114 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw"] Jul 06 19:46:38.447126 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:46:38.447098 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod730a9db9_28ca_4a2e_a069_fd639a1bdddd.slice/crio-c69f25a90634950d6fd0d98b07730881bf78fe7892031580012fc04506647dfe WatchSource:0}: Error finding container c69f25a90634950d6fd0d98b07730881bf78fe7892031580012fc04506647dfe: Status 404 returned error can't find the container with id c69f25a90634950d6fd0d98b07730881bf78fe7892031580012fc04506647dfe Jul 06 19:46:39.004236 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:39.004192 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" event={"ID":"730a9db9-28ca-4a2e-a069-fd639a1bdddd","Type":"ContainerStarted","Data":"eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c"} Jul 06 19:46:39.004675 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:39.004246 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" event={"ID":"730a9db9-28ca-4a2e-a069-fd639a1bdddd","Type":"ContainerStarted","Data":"c69f25a90634950d6fd0d98b07730881bf78fe7892031580012fc04506647dfe"} Jul 06 19:46:39.022121 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:39.022071 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" podStartSLOduration=2.022054926 podStartE2EDuration="2.022054926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:46:39.020806461 +0000 UTC m=+2224.504030053" watchObservedRunningTime="2026-07-06 19:46:39.022054926 +0000 UTC m=+2224.505278499" Jul 06 19:46:42.099559 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:42.099520 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:46:48.311921 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:48.311887 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:48.312442 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:48.311969 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:48.324079 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:48.324046 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:49.062031 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:49.062001 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:46:55.993893 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:46:55.993855 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:47:13.197085 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:13.197050 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:47:21.185273 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:21.185241 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw"] Jul 06 19:47:27.002528 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:27.002495 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:47:34.492284 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.492247 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq"] Jul 06 19:47:34.495837 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.495821 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.498457 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.498434 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-26761c\"/\"kube-root-ca.crt\"" Jul 06 19:47:34.498577 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.498539 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-26761c\"/\"default-dockercfg-t4rpq\"" Jul 06 19:47:34.499828 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.499810 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-26761c\"/\"openshift-service-ca.crt\"" Jul 06 19:47:34.499899 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.499866 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-26761c\"/\"rate-test-model-26761c-kserve-self-signed-certs\"" Jul 06 19:47:34.504531 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.504508 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq"] Jul 06 19:47:34.610039 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.610008 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-model-cache\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.610039 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.610039 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a239d1b-3c86-4922-8184-16e530f271d9-tls-certs\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.610308 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.610059 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mzn\" (UniqueName: \"kubernetes.io/projected/7a239d1b-3c86-4922-8184-16e530f271d9-kube-api-access-z9mzn\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.610308 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.610180 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-dshm\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.610308 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.610212 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-tmp-dir\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.610308 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.610245 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-home\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.711002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.710960 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-model-cache\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.711002 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.711002 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a239d1b-3c86-4922-8184-16e530f271d9-tls-certs\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.711218 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.711028 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mzn\" (UniqueName: \"kubernetes.io/projected/7a239d1b-3c86-4922-8184-16e530f271d9-kube-api-access-z9mzn\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.711218 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.711088 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-dshm\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.711218 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.711115 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-tmp-dir\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.711218 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.711142 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-home\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.711478 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.711456 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-model-cache\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.711544 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.711486 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-tmp-dir\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.711544 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.711517 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-home\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.713437 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.713413 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-dshm\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.713577 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.713560 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a239d1b-3c86-4922-8184-16e530f271d9-tls-certs\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.719140 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.719117 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mzn\" (UniqueName: \"kubernetes.io/projected/7a239d1b-3c86-4922-8184-16e530f271d9-kube-api-access-z9mzn\") pod \"rate-test-model-26761c-kserve-558dfc6dc5-jw7lq\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.806445 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.806350 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:34.938442 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:34.938405 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq"] Jul 06 19:47:34.941875 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:47:34.941849 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a239d1b_3c86_4922_8184_16e530f271d9.slice/crio-6be48f9195f6bef03ba45c5b100bd838f6a8020df1d8d212395082ada1c92589 WatchSource:0}: Error finding container 6be48f9195f6bef03ba45c5b100bd838f6a8020df1d8d212395082ada1c92589: Status 404 returned error can't find the container with id 6be48f9195f6bef03ba45c5b100bd838f6a8020df1d8d212395082ada1c92589 Jul 06 19:47:35.274845 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:35.274806 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" event={"ID":"7a239d1b-3c86-4922-8184-16e530f271d9","Type":"ContainerStarted","Data":"657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911"} Jul 06 19:47:35.274845 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:35.274852 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" event={"ID":"7a239d1b-3c86-4922-8184-16e530f271d9","Type":"ContainerStarted","Data":"6be48f9195f6bef03ba45c5b100bd838f6a8020df1d8d212395082ada1c92589"} Jul 06 19:47:35.294215 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:35.294167 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" podStartSLOduration=1.294153431 podStartE2EDuration="1.294153431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:47:35.293554361 +0000 UTC m=+2280.776777934" watchObservedRunningTime="2026-07-06 19:47:35.294153431 +0000 UTC m=+2280.777377003" Jul 06 19:47:36.205219 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.205155 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" podUID="730a9db9-28ca-4a2e-a069-fd639a1bdddd" containerName="main" containerID="cri-o://eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c" gracePeriod=45 Jul 06 19:47:36.450798 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.450774 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:47:36.527136 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.527042 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh6gd\" (UniqueName: \"kubernetes.io/projected/730a9db9-28ca-4a2e-a069-fd639a1bdddd-kube-api-access-lh6gd\") pod \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " Jul 06 19:47:36.527136 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.527076 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-model-cache\") pod \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " Jul 06 19:47:36.527136 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.527102 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-dshm\") pod \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " Jul 06 19:47:36.527136 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.527123 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tmp-dir\") pod \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " Jul 06 19:47:36.527473 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.527147 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-home\") pod \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " Jul 06 19:47:36.527473 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.527242 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tls-certs\") pod \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\" (UID: \"730a9db9-28ca-4a2e-a069-fd639a1bdddd\") " Jul 06 19:47:36.527585 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.527518 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-home" (OuterVolumeSpecName: "home") pod "730a9db9-28ca-4a2e-a069-fd639a1bdddd" (UID: "730a9db9-28ca-4a2e-a069-fd639a1bdddd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:36.527585 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.527528 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "730a9db9-28ca-4a2e-a069-fd639a1bdddd" (UID: "730a9db9-28ca-4a2e-a069-fd639a1bdddd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:36.527700 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.527587 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-model-cache" (OuterVolumeSpecName: "model-cache") pod "730a9db9-28ca-4a2e-a069-fd639a1bdddd" (UID: "730a9db9-28ca-4a2e-a069-fd639a1bdddd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:36.529594 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.529567 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-dshm" (OuterVolumeSpecName: "dshm") pod "730a9db9-28ca-4a2e-a069-fd639a1bdddd" (UID: "730a9db9-28ca-4a2e-a069-fd639a1bdddd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:36.529594 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.529574 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730a9db9-28ca-4a2e-a069-fd639a1bdddd-kube-api-access-lh6gd" (OuterVolumeSpecName: "kube-api-access-lh6gd") pod "730a9db9-28ca-4a2e-a069-fd639a1bdddd" (UID: "730a9db9-28ca-4a2e-a069-fd639a1bdddd"). InnerVolumeSpecName "kube-api-access-lh6gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:47:36.529782 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.529648 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "730a9db9-28ca-4a2e-a069-fd639a1bdddd" (UID: "730a9db9-28ca-4a2e-a069-fd639a1bdddd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:47:36.628285 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.628258 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh6gd\" (UniqueName: \"kubernetes.io/projected/730a9db9-28ca-4a2e-a069-fd639a1bdddd-kube-api-access-lh6gd\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:47:36.628285 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.628283 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:47:36.628285 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.628292 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:47:36.628509 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.628299 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:47:36.628509 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.628310 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/730a9db9-28ca-4a2e-a069-fd639a1bdddd-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:47:36.628509 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:36.628318 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/730a9db9-28ca-4a2e-a069-fd639a1bdddd-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:47:37.286528 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:37.286494 2554 generic.go:356] "Generic (PLEG): container finished" podID="730a9db9-28ca-4a2e-a069-fd639a1bdddd" containerID="eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c" exitCode=0 Jul 06 19:47:37.286969 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:37.286541 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" Jul 06 19:47:37.286969 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:37.286582 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" event={"ID":"730a9db9-28ca-4a2e-a069-fd639a1bdddd","Type":"ContainerDied","Data":"eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c"} Jul 06 19:47:37.286969 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:37.286629 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw" event={"ID":"730a9db9-28ca-4a2e-a069-fd639a1bdddd","Type":"ContainerDied","Data":"c69f25a90634950d6fd0d98b07730881bf78fe7892031580012fc04506647dfe"} Jul 06 19:47:37.286969 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:37.286649 2554 scope.go:117] "RemoveContainer" containerID="eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c" Jul 06 19:47:37.295471 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:37.295450 2554 scope.go:117] "RemoveContainer" containerID="eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c" Jul 06 19:47:37.295724 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:47:37.295707 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c\": container with ID starting with eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c not found: ID does not exist" containerID="eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c" Jul 06 19:47:37.295831 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:37.295751 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c"} err="failed to get container status \"eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c\": rpc error: code = NotFound desc = could not find container \"eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c\": container with ID starting with eae27234421f07da19b2a03816499c7a978c0b100911f18d9a629e96d6adc80c not found: ID does not exist" Jul 06 19:47:37.296466 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:37.296444 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:47:37.308223 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:37.308195 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw"] Jul 06 19:47:37.311974 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:37.311950 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/sub-test-model-bb8c15-kserve-5f84c94765-hvqsw"] Jul 06 19:47:38.529407 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:38.529372 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="730a9db9-28ca-4a2e-a069-fd639a1bdddd" path="/var/lib/kubelet/pods/730a9db9-28ca-4a2e-a069-fd639a1bdddd/volumes" Jul 06 19:47:44.807187 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:44.807125 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:44.807708 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:44.807277 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:44.819126 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:44.819102 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:45.335787 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:45.335763 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:47:57.898150 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.898115 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4"] Jul 06 19:47:57.898819 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.898671 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="730a9db9-28ca-4a2e-a069-fd639a1bdddd" containerName="main" Jul 06 19:47:57.898819 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.898691 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="730a9db9-28ca-4a2e-a069-fd639a1bdddd" containerName="main" Jul 06 19:47:57.898819 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.898794 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="730a9db9-28ca-4a2e-a069-fd639a1bdddd" containerName="main" Jul 06 19:47:57.901774 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.901753 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:57.905802 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.905778 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"kube-root-ca.crt\"" Jul 06 19:47:57.905917 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.905806 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"openshift-service-ca.crt\"" Jul 06 19:47:57.905917 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.905821 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"default-dockercfg-rvh7f\"" Jul 06 19:47:57.905917 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.905821 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bb8c15\"/\"rate-test-model-bb8c15-kserve-self-signed-certs\"" Jul 06 19:47:57.911434 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.911415 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4"] Jul 06 19:47:57.922697 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.922675 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg8gm\" (UniqueName: \"kubernetes.io/projected/4594b440-1917-4e8b-a622-cc60e2c93a14-kube-api-access-hg8gm\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:57.922824 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.922710 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-tmp-dir\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:57.922824 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.922764 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4594b440-1917-4e8b-a622-cc60e2c93a14-tls-certs\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:57.922824 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.922814 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-home\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:57.922937 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.922859 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-model-cache\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:57.922937 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:57.922904 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-dshm\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.024005 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.023973 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4594b440-1917-4e8b-a622-cc60e2c93a14-tls-certs\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.024005 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.024015 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-home\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.024262 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.024056 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-model-cache\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.024262 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.024082 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-dshm\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.024262 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.024121 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg8gm\" (UniqueName: \"kubernetes.io/projected/4594b440-1917-4e8b-a622-cc60e2c93a14-kube-api-access-hg8gm\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.024262 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.024159 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-tmp-dir\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.024544 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.024521 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-home\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.024610 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.024556 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-tmp-dir\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.024610 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.024590 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-model-cache\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.026369 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.026346 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-dshm\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.026549 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.026532 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4594b440-1917-4e8b-a622-cc60e2c93a14-tls-certs\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.034630 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.034597 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg8gm\" (UniqueName: \"kubernetes.io/projected/4594b440-1917-4e8b-a622-cc60e2c93a14-kube-api-access-hg8gm\") pod \"rate-test-model-bb8c15-kserve-78d7847467-bhrz4\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.212154 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.212065 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:47:58.346910 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.346843 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4"] Jul 06 19:47:58.349315 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:47:58.349275 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4594b440_1917_4e8b_a622_cc60e2c93a14.slice/crio-bc57c280e09a7506b06638aaea9814d162e6bb0b6d1a714644d3c38b804629e9 WatchSource:0}: Error finding container bc57c280e09a7506b06638aaea9814d162e6bb0b6d1a714644d3c38b804629e9: Status 404 returned error can't find the container with id bc57c280e09a7506b06638aaea9814d162e6bb0b6d1a714644d3c38b804629e9 Jul 06 19:47:58.383537 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:58.383506 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" event={"ID":"4594b440-1917-4e8b-a622-cc60e2c93a14","Type":"ContainerStarted","Data":"bc57c280e09a7506b06638aaea9814d162e6bb0b6d1a714644d3c38b804629e9"} Jul 06 19:47:59.389116 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:59.389078 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" event={"ID":"4594b440-1917-4e8b-a622-cc60e2c93a14","Type":"ContainerStarted","Data":"6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa"} Jul 06 19:47:59.409522 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:47:59.409469 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" podStartSLOduration=2.409449651 podStartE2EDuration="2.409449651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:47:59.408000687 +0000 UTC m=+2304.891224283" watchObservedRunningTime="2026-07-06 19:47:59.409449651 +0000 UTC m=+2304.892673225" Jul 06 19:48:01.829968 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:01.829931 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:48:08.212983 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:08.212945 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:48:08.212983 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:08.212987 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:48:08.225918 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:08.225893 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:48:08.444831 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:08.444804 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:48:15.096374 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:15.096335 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:48:29.003220 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:29.003125 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:48:35.179208 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:35.179173 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq"] Jul 06 19:48:36.080234 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:36.080200 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4"] Jul 06 19:48:41.796154 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:41.796116 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:48:50.198297 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.198250 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" podUID="7a239d1b-3c86-4922-8184-16e530f271d9" containerName="main" containerID="cri-o://657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911" gracePeriod=45 Jul 06 19:48:50.441975 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.441951 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:48:50.499197 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499122 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-dshm\") pod \"7a239d1b-3c86-4922-8184-16e530f271d9\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " Jul 06 19:48:50.499327 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499203 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-model-cache\") pod \"7a239d1b-3c86-4922-8184-16e530f271d9\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " Jul 06 19:48:50.499327 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499235 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-home\") pod \"7a239d1b-3c86-4922-8184-16e530f271d9\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " Jul 06 19:48:50.499401 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499324 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9mzn\" (UniqueName: \"kubernetes.io/projected/7a239d1b-3c86-4922-8184-16e530f271d9-kube-api-access-z9mzn\") pod \"7a239d1b-3c86-4922-8184-16e530f271d9\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " Jul 06 19:48:50.499401 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499355 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a239d1b-3c86-4922-8184-16e530f271d9-tls-certs\") pod \"7a239d1b-3c86-4922-8184-16e530f271d9\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " Jul 06 19:48:50.499401 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499393 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-tmp-dir\") pod \"7a239d1b-3c86-4922-8184-16e530f271d9\" (UID: \"7a239d1b-3c86-4922-8184-16e530f271d9\") " Jul 06 19:48:50.499523 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499468 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a239d1b-3c86-4922-8184-16e530f271d9" (UID: "7a239d1b-3c86-4922-8184-16e530f271d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:50.499523 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499483 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-home" (OuterVolumeSpecName: "home") pod "7a239d1b-3c86-4922-8184-16e530f271d9" (UID: "7a239d1b-3c86-4922-8184-16e530f271d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:50.499663 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499646 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a239d1b-3c86-4922-8184-16e530f271d9" (UID: "7a239d1b-3c86-4922-8184-16e530f271d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:50.499663 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499656 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:50.499789 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.499671 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:50.501372 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.501341 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-dshm" (OuterVolumeSpecName: "dshm") pod "7a239d1b-3c86-4922-8184-16e530f271d9" (UID: "7a239d1b-3c86-4922-8184-16e530f271d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:50.501489 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.501380 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a239d1b-3c86-4922-8184-16e530f271d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a239d1b-3c86-4922-8184-16e530f271d9" (UID: "7a239d1b-3c86-4922-8184-16e530f271d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:48:50.501489 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.501444 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a239d1b-3c86-4922-8184-16e530f271d9-kube-api-access-z9mzn" (OuterVolumeSpecName: "kube-api-access-z9mzn") pod "7a239d1b-3c86-4922-8184-16e530f271d9" (UID: "7a239d1b-3c86-4922-8184-16e530f271d9"). InnerVolumeSpecName "kube-api-access-z9mzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:48:50.601087 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.601059 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:50.601087 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.601086 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a239d1b-3c86-4922-8184-16e530f271d9-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:50.601270 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.601097 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9mzn\" (UniqueName: \"kubernetes.io/projected/7a239d1b-3c86-4922-8184-16e530f271d9-kube-api-access-z9mzn\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:50.601270 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.601108 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a239d1b-3c86-4922-8184-16e530f271d9-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:50.623899 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.623865 2554 generic.go:356] "Generic (PLEG): container finished" podID="7a239d1b-3c86-4922-8184-16e530f271d9" containerID="657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911" exitCode=0 Jul 06 19:48:50.624047 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.623921 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" Jul 06 19:48:50.624047 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.623953 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" event={"ID":"7a239d1b-3c86-4922-8184-16e530f271d9","Type":"ContainerDied","Data":"657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911"} Jul 06 19:48:50.624047 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.624012 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq" event={"ID":"7a239d1b-3c86-4922-8184-16e530f271d9","Type":"ContainerDied","Data":"6be48f9195f6bef03ba45c5b100bd838f6a8020df1d8d212395082ada1c92589"} Jul 06 19:48:50.624047 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.624035 2554 scope.go:117] "RemoveContainer" containerID="657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911" Jul 06 19:48:50.632373 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.632357 2554 scope.go:117] "RemoveContainer" containerID="657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911" Jul 06 19:48:50.632621 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:48:50.632604 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911\": container with ID starting with 657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911 not found: ID does not exist" containerID="657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911" Jul 06 19:48:50.632680 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.632630 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911"} err="failed to get container status \"657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911\": rpc error: code = NotFound desc = could not find container \"657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911\": container with ID starting with 657d02a129d9b82a95f004fc363ae20e42d4db71b4772ba258810fdf81309911 not found: ID does not exist" Jul 06 19:48:50.641285 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.641264 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq"] Jul 06 19:48:50.647977 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:50.647955 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-26761c/rate-test-model-26761c-kserve-558dfc6dc5-jw7lq"] Jul 06 19:48:51.100543 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.100483 2554 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" podUID="4594b440-1917-4e8b-a622-cc60e2c93a14" containerName="main" containerID="cri-o://6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa" gracePeriod=45 Jul 06 19:48:51.352767 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.352704 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:48:51.408514 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.408487 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-model-cache\") pod \"4594b440-1917-4e8b-a622-cc60e2c93a14\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " Jul 06 19:48:51.408683 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.408532 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-tmp-dir\") pod \"4594b440-1917-4e8b-a622-cc60e2c93a14\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " Jul 06 19:48:51.408683 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.408584 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg8gm\" (UniqueName: \"kubernetes.io/projected/4594b440-1917-4e8b-a622-cc60e2c93a14-kube-api-access-hg8gm\") pod \"4594b440-1917-4e8b-a622-cc60e2c93a14\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " Jul 06 19:48:51.408683 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.408605 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4594b440-1917-4e8b-a622-cc60e2c93a14-tls-certs\") pod \"4594b440-1917-4e8b-a622-cc60e2c93a14\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " Jul 06 19:48:51.408683 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.408621 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-dshm\") pod \"4594b440-1917-4e8b-a622-cc60e2c93a14\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " Jul 06 19:48:51.408683 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.408678 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-home\") pod \"4594b440-1917-4e8b-a622-cc60e2c93a14\" (UID: \"4594b440-1917-4e8b-a622-cc60e2c93a14\") " Jul 06 19:48:51.408997 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.408817 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-model-cache" (OuterVolumeSpecName: "model-cache") pod "4594b440-1917-4e8b-a622-cc60e2c93a14" (UID: "4594b440-1917-4e8b-a622-cc60e2c93a14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:51.408997 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.408971 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4594b440-1917-4e8b-a622-cc60e2c93a14" (UID: "4594b440-1917-4e8b-a622-cc60e2c93a14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:51.409083 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.408995 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-home" (OuterVolumeSpecName: "home") pod "4594b440-1917-4e8b-a622-cc60e2c93a14" (UID: "4594b440-1917-4e8b-a622-cc60e2c93a14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:51.409083 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.409050 2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:51.410646 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.410615 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-dshm" (OuterVolumeSpecName: "dshm") pod "4594b440-1917-4e8b-a622-cc60e2c93a14" (UID: "4594b440-1917-4e8b-a622-cc60e2c93a14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:51.410806 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.410785 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4594b440-1917-4e8b-a622-cc60e2c93a14-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4594b440-1917-4e8b-a622-cc60e2c93a14" (UID: "4594b440-1917-4e8b-a622-cc60e2c93a14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:48:51.410856 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.410798 2554 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4594b440-1917-4e8b-a622-cc60e2c93a14-kube-api-access-hg8gm" (OuterVolumeSpecName: "kube-api-access-hg8gm") pod "4594b440-1917-4e8b-a622-cc60e2c93a14" (UID: "4594b440-1917-4e8b-a622-cc60e2c93a14"). InnerVolumeSpecName "kube-api-access-hg8gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:48:51.509684 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.509650 2554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-home\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:51.509684 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.509676 2554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-tmp-dir\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:51.509684 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.509687 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg8gm\" (UniqueName: \"kubernetes.io/projected/4594b440-1917-4e8b-a622-cc60e2c93a14-kube-api-access-hg8gm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:51.509936 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.509695 2554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4594b440-1917-4e8b-a622-cc60e2c93a14-tls-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:51.509936 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.509704 2554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4594b440-1917-4e8b-a622-cc60e2c93a14-dshm\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Jul 06 19:48:51.631331 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.631232 2554 generic.go:356] "Generic (PLEG): container finished" podID="4594b440-1917-4e8b-a622-cc60e2c93a14" containerID="6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa" exitCode=0 Jul 06 19:48:51.631331 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.631279 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" Jul 06 19:48:51.631331 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.631300 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" event={"ID":"4594b440-1917-4e8b-a622-cc60e2c93a14","Type":"ContainerDied","Data":"6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa"} Jul 06 19:48:51.631617 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.631338 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4" event={"ID":"4594b440-1917-4e8b-a622-cc60e2c93a14","Type":"ContainerDied","Data":"bc57c280e09a7506b06638aaea9814d162e6bb0b6d1a714644d3c38b804629e9"} Jul 06 19:48:51.631617 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.631360 2554 scope.go:117] "RemoveContainer" containerID="6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa" Jul 06 19:48:51.640137 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.640120 2554 scope.go:117] "RemoveContainer" containerID="6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa" Jul 06 19:48:51.640372 ip-10-0-142-236 kubenswrapper[2554]: E0706 19:48:51.640356 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa\": container with ID starting with 6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa not found: ID does not exist" containerID="6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa" Jul 06 19:48:51.640429 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.640383 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa"} err="failed to get container status \"6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa\": rpc error: code = NotFound desc = could not find container \"6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa\": container with ID starting with 6cf96a8c74a3959135e1c92fc02161986c17aa0e5802743c4fc20d13c582feaa not found: ID does not exist" Jul 06 19:48:51.652194 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.652165 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4"] Jul 06 19:48:51.656635 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:51.656613 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bb8c15/rate-test-model-bb8c15-kserve-78d7847467-bhrz4"] Jul 06 19:48:52.530250 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:52.530216 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4594b440-1917-4e8b-a622-cc60e2c93a14" path="/var/lib/kubelet/pods/4594b440-1917-4e8b-a622-cc60e2c93a14/volumes" Jul 06 19:48:52.530621 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:48:52.530609 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a239d1b-3c86-4922-8184-16e530f271d9" path="/var/lib/kubelet/pods/7a239d1b-3c86-4922-8184-16e530f271d9/volumes" Jul 06 19:49:24.093090 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:24.093054 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7jjw8"] Jul 06 19:49:27.666775 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:27.666744 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f97c9fc9c-d96hp_84d4deb3-8dea-4149-89f9-81408f9da6fd/authorino/0.log" Jul 06 19:49:32.414938 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:32.414908 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dcwfg_85d5ae65-b3cb-41ec-859e-1ec405a2e877/manager/0.log" Jul 06 19:49:32.646309 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:32.646282 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722770-gzxl4_4a7d4507-b124-4813-8a96-6e86b6b0a4f8/cleanup/0.log" Jul 06 19:49:33.346674 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:33.346648 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c47dfb7f-vgtnr_d32cff0c-e6d9-456a-b294-00d2d6db5fad/copy-manifests/0.log" Jul 06 19:49:33.353129 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:33.353103 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c47dfb7f-vgtnr_d32cff0c-e6d9-456a-b294-00d2d6db5fad/manager/0.log" Jul 06 19:49:34.867182 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:34.867151 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f97c9fc9c-d96hp_84d4deb3-8dea-4149-89f9-81408f9da6fd/authorino/0.log" Jul 06 19:49:35.549432 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:35.549399 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7jjw8_ff816c7b-3f56-4364-8419-d4d11089abcc/limitador/0.log" Jul 06 19:49:36.262772 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:36.262714 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-754576df6f-8mtfg_126eac73-4dcc-4849-8744-8d42cfe8db14/kube-auth-proxy/0.log" Jul 06 19:49:36.589103 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:36.589075 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:49:36.593628 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:36.593608 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bfz6c_d8724e53-459c-4708-aa69-9ca66d0f90ab/ovn-acl-logging/0.log" Jul 06 19:49:37.304572 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:37.304542 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5w7xv_80740e3d-ddb4-4c31-9a71-870a1b09acfd/main/0.log" Jul 06 19:49:45.245259 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:45.245231 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ncc6d_423a3192-16a1-4e66-94de-9d88c359469c/global-pull-secret-syncer/0.log" Jul 06 19:49:45.408379 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:45.408349 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x2dvn_dd8720ca-7ec0-425b-9b19-2afaf1699d4d/konnectivity-agent/0.log" Jul 06 19:49:45.477535 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:45.477508 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-236.ec2.internal_69103f5be0b7c8797e5968b28b95762b/haproxy/0.log" Jul 06 19:49:50.080161 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:50.080124 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f97c9fc9c-d96hp_84d4deb3-8dea-4149-89f9-81408f9da6fd/authorino/0.log" Jul 06 19:49:50.425860 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:50.425696 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7jjw8_ff816c7b-3f56-4364-8419-d4d11089abcc/limitador/0.log" Jul 06 19:49:52.219402 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.219374 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6d5sm_4d54b830-a51c-4979-b181-4c47e5988b07/kube-state-metrics/0.log" Jul 06 19:49:52.241688 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.241659 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6d5sm_4d54b830-a51c-4979-b181-4c47e5988b07/kube-rbac-proxy-main/0.log" Jul 06 19:49:52.262595 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.262569 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6d5sm_4d54b830-a51c-4979-b181-4c47e5988b07/kube-rbac-proxy-self/0.log" Jul 06 19:49:52.293568 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.293508 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-55495fc5d5-fpvk6_b6e1ac1c-7d0a-4306-ad97-2639ab10ed81/metrics-server/0.log" Jul 06 19:49:52.319621 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.319593 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-6bgpn_e7cffb20-d0f2-4323-bc35-e5695a8339f0/monitoring-plugin/0.log" Jul 06 19:49:52.430344 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.430311 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-874h4_f039c33b-9c32-4295-a96a-4526f3de292d/node-exporter/0.log" Jul 06 19:49:52.452546 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.452519 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-874h4_f039c33b-9c32-4295-a96a-4526f3de292d/kube-rbac-proxy/0.log" Jul 06 19:49:52.476315 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.476291 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-874h4_f039c33b-9c32-4295-a96a-4526f3de292d/init-textfile/0.log" Jul 06 19:49:52.591215 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.591188 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-m6vd4_d21596dd-12ac-4c85-aeff-ac0b60181817/kube-rbac-proxy-main/0.log" Jul 06 19:49:52.617610 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.617588 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-m6vd4_d21596dd-12ac-4c85-aeff-ac0b60181817/kube-rbac-proxy-self/0.log" Jul 06 19:49:52.643724 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.643703 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-m6vd4_d21596dd-12ac-4c85-aeff-ac0b60181817/openshift-state-metrics/0.log" Jul 06 19:49:52.697610 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.697585 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79aa8a2b-7643-4a49-b603-57fb45202849/prometheus/0.log" Jul 06 19:49:52.720044 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.720012 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79aa8a2b-7643-4a49-b603-57fb45202849/config-reloader/0.log" Jul 06 19:49:52.740217 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.740194 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79aa8a2b-7643-4a49-b603-57fb45202849/thanos-sidecar/0.log" Jul 06 19:49:52.764366 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.764342 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79aa8a2b-7643-4a49-b603-57fb45202849/kube-rbac-proxy-web/0.log" Jul 06 19:49:52.786046 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.786019 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79aa8a2b-7643-4a49-b603-57fb45202849/kube-rbac-proxy/0.log" Jul 06 19:49:52.807622 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.807593 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79aa8a2b-7643-4a49-b603-57fb45202849/kube-rbac-proxy-thanos/0.log" Jul 06 19:49:52.828127 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.828107 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_79aa8a2b-7643-4a49-b603-57fb45202849/init-config-reloader/0.log" Jul 06 19:49:52.913218 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:52.913193 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-vng67_699f4628-62ca-48d0-b7fa-4ac5b289b9a8/prometheus-operator-admission-webhook/0.log" Jul 06 19:49:54.519258 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:54.519230 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-w4tgb_af68ec4b-e432-4f9e-92e5-4ec15b9abb3d/networking-console-plugin/0.log" Jul 06 19:49:55.617623 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.617583 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p"] Jul 06 19:49:55.618022 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.617955 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a239d1b-3c86-4922-8184-16e530f271d9" containerName="main" Jul 06 19:49:55.618022 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.617967 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a239d1b-3c86-4922-8184-16e530f271d9" containerName="main" Jul 06 19:49:55.618022 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.617975 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4594b440-1917-4e8b-a622-cc60e2c93a14" containerName="main" Jul 06 19:49:55.618022 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.617979 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4594b440-1917-4e8b-a622-cc60e2c93a14" containerName="main" Jul 06 19:49:55.618285 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.618035 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4594b440-1917-4e8b-a622-cc60e2c93a14" containerName="main" Jul 06 19:49:55.618285 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.618076 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a239d1b-3c86-4922-8184-16e530f271d9" containerName="main" Jul 06 19:49:55.621342 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.621322 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.624053 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.624027 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p6v6j\"/\"openshift-service-ca.crt\"" Jul 06 19:49:55.624171 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.624114 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p6v6j\"/\"kube-root-ca.crt\"" Jul 06 19:49:55.624322 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.624308 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p6v6j\"/\"default-dockercfg-f9kr7\"" Jul 06 19:49:55.631013 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.630988 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p"] Jul 06 19:49:55.644238 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.644217 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74ff6b4789-sjf5k_7f5c83ef-00b1-463c-9622-a0641cad5d08/console/0.log" Jul 06 19:49:55.674447 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.674422 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-ktkpf_fabc8a56-c8b0-4def-9e61-5b5430ac4538/download-server/0.log" Jul 06 19:49:55.754215 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.754179 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-lib-modules\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.754388 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.754239 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-proc\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.754388 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.754291 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-podres\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.754388 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.754305 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-sys\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.754388 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.754359 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hgd\" (UniqueName: \"kubernetes.io/projected/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-kube-api-access-24hgd\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.854806 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.854770 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-podres\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.854806 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.854805 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-sys\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.855014 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.854826 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hgd\" (UniqueName: \"kubernetes.io/projected/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-kube-api-access-24hgd\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.855014 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.854863 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-lib-modules\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.855014 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.854915 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-sys\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.855014 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.854936 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-proc\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.855014 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.854961 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-podres\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.855171 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.855018 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-proc\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.855171 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.855046 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-lib-modules\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.862874 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.862844 2554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hgd\" (UniqueName: \"kubernetes.io/projected/d1f53a13-8a4e-4f76-8bc5-b1fa962ed907-kube-api-access-24hgd\") pod \"perf-node-gather-daemonset-7mn2p\" (UID: \"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907\") " pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:55.932069 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:55.931981 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:56.060603 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:56.060560 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p"] Jul 06 19:49:56.062843 ip-10-0-142-236 kubenswrapper[2554]: W0706 19:49:56.062812 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd1f53a13_8a4e_4f76_8bc5_b1fa962ed907.slice/crio-2ce4b1d859be2b91f3dadec6ba5618623a0fee5b6683dcd1bca9c101c39aa7ee WatchSource:0}: Error finding container 2ce4b1d859be2b91f3dadec6ba5618623a0fee5b6683dcd1bca9c101c39aa7ee: Status 404 returned error can't find the container with id 2ce4b1d859be2b91f3dadec6ba5618623a0fee5b6683dcd1bca9c101c39aa7ee Jul 06 19:49:56.932844 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:56.932755 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" event={"ID":"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907","Type":"ContainerStarted","Data":"f82d56114183d38d32461cdd11886ebfe04858d9ac2122260a383216e67a656e"} Jul 06 19:49:56.932844 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:56.932804 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" event={"ID":"d1f53a13-8a4e-4f76-8bc5-b1fa962ed907","Type":"ContainerStarted","Data":"2ce4b1d859be2b91f3dadec6ba5618623a0fee5b6683dcd1bca9c101c39aa7ee"} Jul 06 19:49:56.933242 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:56.932841 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:49:56.950072 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:56.950027 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" podStartSLOduration=1.950010059 podStartE2EDuration="1.950010059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:49:56.948367398 +0000 UTC m=+2422.431590972" watchObservedRunningTime="2026-07-06 19:49:56.950010059 +0000 UTC m=+2422.433233631" Jul 06 19:49:57.332246 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:57.332210 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9x7qx_db3da481-af3b-4827-95e5-bdd08e4ff256/dns/0.log" Jul 06 19:49:57.352362 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:57.352334 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9x7qx_db3da481-af3b-4827-95e5-bdd08e4ff256/kube-rbac-proxy/0.log" Jul 06 19:49:57.483776 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:57.483749 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-56v7k_c1c2a219-c8c6-4937-952e-630b3eb7de5a/dns-node-resolver/0.log" Jul 06 19:49:58.076502 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:58.076460 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xqhsh_eaebfdaf-a012-490a-8e17-b8346cbf6748/node-ca/0.log" Jul 06 19:49:59.166721 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:49:59.166682 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-754576df6f-8mtfg_126eac73-4dcc-4849-8744-8d42cfe8db14/kube-auth-proxy/0.log" Jul 06 19:50:00.019382 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:00.019352 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fpfnt_ae485556-28dc-4ad6-98e2-7b41753bb66e/serve-healthcheck-canary/0.log" Jul 06 19:50:00.515086 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:00.515053 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-58r22_11ca926e-8858-4b62-b157-62833f8411e1/insights-operator/0.log" Jul 06 19:50:00.515560 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:00.515317 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-58r22_11ca926e-8858-4b62-b157-62833f8411e1/insights-operator/1.log" Jul 06 19:50:00.658853 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:00.658824 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jljqf_60469931-e60f-4ff2-958c-147f23e5c775/kube-rbac-proxy/0.log" Jul 06 19:50:00.678233 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:00.678206 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jljqf_60469931-e60f-4ff2-958c-147f23e5c775/exporter/0.log" Jul 06 19:50:00.697576 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:00.697551 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jljqf_60469931-e60f-4ff2-958c-147f23e5c775/extractor/0.log" Jul 06 19:50:02.885692 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:02.885663 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-dcwfg_85d5ae65-b3cb-41ec-859e-1ec405a2e877/manager/0.log" Jul 06 19:50:02.950337 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:02.950306 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p6v6j/perf-node-gather-daemonset-7mn2p" Jul 06 19:50:02.956217 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:02.956196 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722770-gzxl4_4a7d4507-b124-4813-8a96-6e86b6b0a4f8/cleanup/0.log" Jul 06 19:50:03.609078 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:03.609046 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c47dfb7f-vgtnr_d32cff0c-e6d9-456a-b294-00d2d6db5fad/manager/0.log" Jul 06 19:50:03.643387 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:03.643362 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55c47dfb7f-vgtnr_d32cff0c-e6d9-456a-b294-00d2d6db5fad/copy-manifests/0.log" Jul 06 19:50:05.369487 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:05.369460 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78958bf977-j5rxz_6dfcb6ba-9dbd-4559-884e-a31a3989de13/manager/0.log" Jul 06 19:50:05.613045 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:05.612970 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-754576df6f-8mtfg_126eac73-4dcc-4849-8744-8d42cfe8db14/kube-auth-proxy/0.log" Jul 06 19:50:07.201477 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:07.201441 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-754576df6f-8mtfg_126eac73-4dcc-4849-8744-8d42cfe8db14/kube-auth-proxy/0.log" Jul 06 19:50:10.713655 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:10.713617 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-mdnbf_cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a/kube-storage-version-migrator-operator/1.log" Jul 06 19:50:10.714453 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:10.714434 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-mdnbf_cd0881a9-cdc6-4fd5-928c-1d5c9030cc4a/kube-storage-version-migrator-operator/0.log" Jul 06 19:50:12.085006 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:12.084978 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xlh9g_598c2c71-e257-4445-8f36-710b469c366f/kube-multus-additional-cni-plugins/0.log" Jul 06 19:50:12.106035 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:12.106010 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xlh9g_598c2c71-e257-4445-8f36-710b469c366f/egress-router-binary-copy/0.log" Jul 06 19:50:12.129523 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:12.129502 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xlh9g_598c2c71-e257-4445-8f36-710b469c366f/cni-plugins/0.log" Jul 06 19:50:12.151636 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:12.151614 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xlh9g_598c2c71-e257-4445-8f36-710b469c366f/bond-cni-plugin/0.log" Jul 06 19:50:12.171640 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:12.171619 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xlh9g_598c2c71-e257-4445-8f36-710b469c366f/routeoverride-cni/0.log" Jul 06 19:50:12.192531 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:12.192512 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xlh9g_598c2c71-e257-4445-8f36-710b469c366f/whereabouts-cni-bincopy/0.log" Jul 06 19:50:12.213422 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:12.213402 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xlh9g_598c2c71-e257-4445-8f36-710b469c366f/whereabouts-cni/0.log" Jul 06 19:50:12.278813 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:12.278783 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kj27t_b89320cc-3dc2-406c-aa36-62aa5e8f377b/kube-multus/0.log" Jul 06 19:50:12.385614 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:12.385533 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hlrkz_ee50f9b8-fc16-4951-a486-88d460005a0c/network-metrics-daemon/0.log" Jul 06 19:50:12.407291 ip-10-0-142-236 kubenswrapper[2554]: I0706 19:50:12.407258 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hlrkz_ee50f9b8-fc16-4951-a486-88d460005a0c/kube-rbac-proxy/0.log"