Jul 17 18:33:02.976814 ip-10-0-132-220 systemd[1]: Starting Kubernetes Kubelet... Jul 17 18:33:03.381908 ip-10-0-132-220 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:33:03.381908 ip-10-0-132-220 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 18:33:03.381908 ip-10-0-132-220 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:33:03.381908 ip-10-0-132-220 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 18:33:03.381908 ip-10-0-132-220 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 18:33:03.383158 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.382773 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 18:33:03.387825 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387801 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:33:03.387825 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387824 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:33:03.387825 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387827 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:33:03.387825 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387831 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387834 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387837 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387840 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387843 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387846 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387849 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387854 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387858 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387861 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387864 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387866 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387869 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387872 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387875 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387877 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387880 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387883 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387885 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387888 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:33:03.387961 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387891 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387894 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387896 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387899 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387902 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387906 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387910 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387914 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387917 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387920 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387923 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387926 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387928 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387931 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387934 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387937 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387940 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387943 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387946 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:33:03.388438 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387948 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387951 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387953 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387956 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387958 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387961 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387967 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387970 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387975 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387979 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387982 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387984 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387987 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387989 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387992 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387994 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.387997 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388000 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388003 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:33:03.388918 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388005 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388008 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388010 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388013 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388016 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388019 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388022 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388025 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388028 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388031 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388034 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388037 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388039 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388044 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388047 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388049 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388052 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388055 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388057 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388059 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:33:03.389372 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388062 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388065 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388067 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388070 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388072 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388075 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388077 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388080 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388083 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388085 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388089 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388092 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388095 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388097 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388100 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388102 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388105 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388108 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388842 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388850 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:33:03.389917 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388853 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388857 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388859 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388863 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388865 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388868 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388870 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388873 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388876 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388878 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388881 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388884 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388886 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388889 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388891 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388894 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388896 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388901 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388905 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388908 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:33:03.390397 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388911 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388914 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388918 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388920 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388924 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388926 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388930 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388933 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388936 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388939 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388943 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388946 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388949 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388952 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388955 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388958 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388961 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388965 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388968 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:33:03.391017 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388971 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388974 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388976 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388979 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388982 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388984 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388987 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388989 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388992 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388995 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.388998 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389000 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389003 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389005 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389008 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389010 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389013 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389017 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389020 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:33:03.391517 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389023 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389027 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389030 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389033 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389036 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389038 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389041 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389044 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389046 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389049 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389052 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389055 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389058 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389060 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389062 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389065 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389068 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389070 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389072 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389075 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:33:03.391982 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389077 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389080 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389083 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389085 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389088 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389091 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389093 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389096 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389099 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389103 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389105 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389108 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389110 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389113 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389116 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389118 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389121 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389123 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389126 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389535 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389540 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:33:03.392506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389543 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389546 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389553 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389556 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389558 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389561 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389564 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389566 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389569 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389572 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389574 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389577 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389580 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389583 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389586 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389589 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389591 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389594 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389597 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:33:03.393028 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389599 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389602 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389605 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389607 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389611 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389615 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389618 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389622 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389625 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389628 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389630 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389633 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389636 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389638 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389641 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389644 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389647 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389649 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389652 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:33:03.393525 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389655 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389657 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389660 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389663 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389665 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389671 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389674 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389676 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389679 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389681 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389684 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389687 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389689 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389692 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389694 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389697 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389700 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389702 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389704 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389708 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:33:03.393991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389710 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389714 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389717 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389719 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389723 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389727 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389730 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389732 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389735 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389737 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389740 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389743 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389746 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389748 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389750 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389753 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389756 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389761 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389763 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:33:03.394498 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389766 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389768 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389771 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389773 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389776 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389779 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389781 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389783 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389786 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389789 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389791 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389794 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389797 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389799 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389802 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389804 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389807 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389809 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389812 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.389814 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:33:03.395007 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389892 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389900 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389907 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389911 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389917 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389920 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389925 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389930 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389933 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389936 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389942 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389945 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389948 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389951 2563 flags.go:64] FLAG: --cgroup-root="" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389954 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389957 2563 flags.go:64] FLAG: --client-ca-file="" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389960 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389962 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389967 2563 flags.go:64] FLAG: --cluster-domain="" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389970 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389974 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389977 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389981 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389985 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 18:33:03.395513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389989 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389992 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389995 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.389998 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390001 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390004 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390008 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390011 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390015 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390017 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390021 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390024 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390027 2563 flags.go:64] FLAG: --enable-server="true" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390030 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390035 2563 flags.go:64] FLAG: --event-burst="100" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390038 2563 flags.go:64] FLAG: --event-qps="50" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390041 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390044 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390048 2563 flags.go:64] FLAG: --eviction-hard="" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390052 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390055 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390058 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390061 2563 flags.go:64] FLAG: --eviction-soft="" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390063 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390066 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 18:33:03.396086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390069 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390072 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390075 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390078 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390081 2563 flags.go:64] FLAG: --feature-gates="" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390084 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390088 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390091 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390094 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390097 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390100 2563 flags.go:64] FLAG: --help="false" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390103 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390106 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390109 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390112 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390115 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390119 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390122 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390125 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390127 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390130 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390133 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390137 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390139 2563 flags.go:64] FLAG: --kube-reserved="" Jul 17 18:33:03.396716 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390142 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390147 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390150 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390153 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390155 2563 flags.go:64] FLAG: --lock-file="" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390158 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390161 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390164 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390169 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390172 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390175 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390177 2563 flags.go:64] FLAG: --logging-format="text" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390180 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390183 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390186 2563 flags.go:64] FLAG: --manifest-url="" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390189 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390194 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390197 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390201 2563 flags.go:64] FLAG: --max-pods="110" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390204 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390207 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390210 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390212 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390215 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390218 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 18:33:03.397320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390221 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390229 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390232 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390234 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390238 2563 flags.go:64] FLAG: --pod-cidr="" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390241 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390247 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390250 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390255 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390258 2563 flags.go:64] FLAG: --port="10250" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390261 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390264 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09482c33eba9e2f5b" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390267 2563 flags.go:64] FLAG: --qos-reserved="" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390270 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390272 2563 flags.go:64] FLAG: --register-node="true" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390275 2563 flags.go:64] FLAG: --register-with-taints="" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390279 2563 flags.go:64] FLAG: --registry-burst="10" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390282 2563 flags.go:64] FLAG: --registry-qps="5" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390285 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390288 2563 flags.go:64] FLAG: --reserved-memory="" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390292 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390295 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390298 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390301 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390303 2563 flags.go:64] FLAG: --runonce="false" Jul 17 18:33:03.397953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390306 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390309 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390312 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390315 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390317 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390320 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390323 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390328 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390330 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390333 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390336 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390339 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390342 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390345 2563 flags.go:64] FLAG: --system-cgroups="" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390348 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390355 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390358 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390361 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390366 2563 flags.go:64] FLAG: --tls-min-version="" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390368 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390371 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390374 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390377 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390380 2563 flags.go:64] FLAG: --v="2" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390385 2563 flags.go:64] FLAG: --version="false" Jul 17 18:33:03.398613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390390 2563 flags.go:64] FLAG: --vmodule="" Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390394 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.390397 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390532 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390537 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390540 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390543 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390547 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390550 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390553 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390556 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390558 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390561 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390563 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390566 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390569 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390572 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390576 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390580 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390582 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:33:03.399228 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390585 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390588 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390592 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390595 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390599 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390602 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390605 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390608 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390611 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390613 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390616 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390621 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390624 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390626 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390630 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390632 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390635 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390638 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390640 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:33:03.399740 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390643 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390647 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390650 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390653 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390656 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390659 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390661 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390664 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390666 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390669 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390672 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390675 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390677 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390680 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390682 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390686 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390688 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390691 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390693 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:33:03.400217 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390696 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390698 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390701 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390703 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390706 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390710 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390712 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390715 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390717 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390720 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390723 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390725 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390728 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390731 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390734 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390736 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390739 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390742 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390744 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:33:03.400705 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390747 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390749 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390752 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390755 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390758 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390760 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390763 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390765 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390768 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390771 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390774 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390776 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390779 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390781 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390784 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390787 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390789 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390791 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390795 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390798 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:33:03.401184 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390801 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390803 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390806 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390808 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.390811 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.391727 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.399403 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.399430 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.399462 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.399474 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399545 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399551 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399554 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399557 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:33:03.401695 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399560 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399562 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399565 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399567 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399570 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399572 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399575 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399578 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399580 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399583 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399585 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399588 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399590 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399593 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399595 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399598 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399600 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399603 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399605 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399608 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:33:03.402055 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399610 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399612 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399615 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399618 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399621 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399630 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399633 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399636 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399638 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399641 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399643 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399646 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399648 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399651 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399653 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399656 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399660 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399671 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399673 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:33:03.402565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399676 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399679 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399681 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399684 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399687 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399689 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399692 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399694 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399697 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399699 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399702 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399704 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399707 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399709 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399713 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399716 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399718 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399721 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399724 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399732 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:33:03.403032 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399735 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399738 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399740 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399743 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399745 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399748 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399750 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399753 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399756 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399758 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399760 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399763 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399765 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399768 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399770 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399772 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399775 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399777 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399780 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399782 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:33:03.403646 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399785 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399789 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399792 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399795 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399797 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399800 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399803 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399806 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399808 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399811 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399813 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399816 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399824 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399828 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399832 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:33:03.404126 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.399835 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.399841 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400011 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400018 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400021 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400024 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400027 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400030 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400033 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400036 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400038 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400042 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400046 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400049 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400052 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 18:33:03.404506 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400055 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400057 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400060 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400062 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400065 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400067 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400070 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400072 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400075 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400077 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400080 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400082 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400086 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400088 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400091 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400101 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400104 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400106 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400109 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400111 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 18:33:03.404871 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400114 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400116 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400119 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400121 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400124 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400126 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400129 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400131 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400134 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400137 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400139 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400142 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400144 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400147 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400149 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400152 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400154 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400157 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400159 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400162 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 18:33:03.405358 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400164 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400167 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400170 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400172 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400175 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400178 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400180 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400182 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400191 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400193 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400196 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400200 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400203 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400205 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400208 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400210 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400214 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400218 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400221 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 18:33:03.405869 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400224 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400227 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400230 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400232 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400235 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400237 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400240 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400242 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400245 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400247 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400249 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400252 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400255 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400257 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400260 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400262 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400265 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400268 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400271 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400273 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 18:33:03.406334 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400275 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400278 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400286 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400289 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400291 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400293 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:03.400296 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.400301 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.400477 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.402716 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.404269 2563 server.go:1019] "Starting client certificate rotation" Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.404371 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 18:33:03.407182 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.404948 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 18:33:03.408749 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.408726 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 18:33:03.410206 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.410180 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 18:33:03.422719 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.422687 2563 log.go:25] "Validated CRI v1 runtime API" Jul 17 18:33:03.427292 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.427274 2563 log.go:25] "Validated CRI v1 image API" Jul 17 18:33:03.428300 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.428278 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 18:33:03.433043 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.433014 2563 fs.go:135] Filesystem UUIDs: map[55feaf29-a566-4610-ae6c-2d82e8492991:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b01c24f8-2a22-4eed-b0f2-e43aaeb10e3a:/dev/nvme0n1p4] Jul 17 18:33:03.433118 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.433043 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 18:33:03.439517 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.439357 2563 manager.go:216] Machine: {Timestamp:2026-07-17 18:33:03.437338072 +0000 UTC m=+0.340911939 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2824757 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e63d045ce3f376936e6dc1b5bd184 SystemUUID:ec2e63d0-45ce-3f37-6936-e6dc1b5bd184 BootID:e3883606-de90-485b-a139-3c06ae5cae8c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e8:59:23:dd:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e8:59:23:dd:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:6c:e1:6d:aa:4d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 18:33:03.439517 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.439510 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 18:33:03.439677 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.439659 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 18:33:03.440792 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.440752 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 18:33:03.440962 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.440795 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-220.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 18:33:03.442093 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.442080 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 18:33:03.442133 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.442097 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 18:33:03.442133 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.442110 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 18:33:03.442803 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.442790 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 18:33:03.444085 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.444070 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 18:33:03.444315 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.444295 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 18:33:03.447118 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.447101 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 17 18:33:03.447274 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.447264 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 18:33:03.447314 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.447304 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 17 18:33:03.447345 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.447321 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 18:33:03.447345 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.447341 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 18:33:03.447396 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.447356 2563 kubelet.go:398] "Adding apiserver pod source" Jul 17 18:33:03.447396 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.447369 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 18:33:03.447814 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.447801 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 18:33:03.447857 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.447821 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 18:33:03.451336 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.451307 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 18:33:03.453732 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.453707 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 18:33:03.453847 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.453761 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 18:33:03.454953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.454939 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 18:33:03.455000 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.454964 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 18:33:03.455000 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.454984 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 18:33:03.455000 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.454993 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 18:33:03.455083 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.455005 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 18:33:03.455083 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.455011 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 18:33:03.455083 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.455020 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 18:33:03.455083 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.455026 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 18:33:03.455083 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.455034 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 18:33:03.455083 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.455040 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 18:33:03.455083 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.455049 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 18:33:03.455083 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.455055 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 18:33:03.455765 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.455753 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 18:33:03.455802 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.455767 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 18:33:03.459565 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.459550 2563 server.go:1268] "Started kubelet" Jul 17 18:33:03.459665 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.459632 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 18:33:03.459724 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.459654 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 18:33:03.459797 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.459737 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 18:33:03.460850 ip-10-0-132-220 systemd[1]: Started Kubernetes Kubelet. Jul 17 18:33:03.462759 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.462735 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-220.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 18:33:03.462852 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.462840 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 17 18:33:03.463221 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.463173 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-220.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 18:33:03.463221 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.463183 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 18:33:03.463469 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.463431 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 18:33:03.467482 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.467437 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 18:33:03.469572 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.469548 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 18:33:03.469665 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.469577 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 18:33:03.469763 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.469746 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:03.470136 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.470116 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 18:33:03.470225 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.470118 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 18:33:03.470225 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.470157 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 18:33:03.470323 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.470288 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 17 18:33:03.470323 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.470298 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 17 18:33:03.470512 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.470492 2563 factory.go:55] Registering systemd factory Jul 17 18:33:03.470512 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.470518 2563 factory.go:223] Registration of the systemd container factory successfully Jul 17 18:33:03.470660 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.468559 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-220.ec2.internal.18c32762fc2694b1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-220.ec2.internal,UID:ip-10-0-132-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-220.ec2.internal,},FirstTimestamp:2026-07-17 18:33:03.459513521 +0000 UTC m=+0.363087387,LastTimestamp:2026-07-17 18:33:03.459513521 +0000 UTC m=+0.363087387,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-220.ec2.internal,}" Jul 17 18:33:03.470754 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.470743 2563 factory.go:153] Registering CRI-O factory Jul 17 18:33:03.470810 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.470758 2563 factory.go:223] Registration of the crio container factory successfully Jul 17 18:33:03.471276 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.470809 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 18:33:03.471336 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.471289 2563 factory.go:103] Registering Raw factory Jul 17 18:33:03.471336 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.471300 2563 manager.go:1196] Started watching for new ooms in manager Jul 17 18:33:03.471760 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.471744 2563 manager.go:349] Starting recovery of all containers Jul 17 18:33:03.480170 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.479933 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 18:33:03.480727 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.480697 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-220.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 18:33:03.480897 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.480867 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 18:33:03.485041 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.485016 2563 manager.go:354] Recovery completed Jul 17 18:33:03.489525 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.489506 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:33:03.491334 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.491312 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:33:03.491421 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.491352 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:33:03.491421 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.491367 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:33:03.491968 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.491952 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 18:33:03.491968 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.491967 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 18:33:03.492074 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.491983 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 17 18:33:03.492074 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.491988 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:33:03.492801 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.492785 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:33:03.492871 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.492818 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:33:03.492871 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.492832 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:33:03.493282 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.493270 2563 policy_none.go:49] "None policy: Start" Jul 17 18:33:03.493312 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.493288 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 18:33:03.493312 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.493299 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 18:33:03.494322 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.494307 2563 policy_none.go:47] "Start" Jul 17 18:33:03.494438 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.494350 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-220.ec2.internal.18c32762fe0c1e08 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-220.ec2.internal,UID:ip-10-0-132-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-220.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-220.ec2.internal,},FirstTimestamp:2026-07-17 18:33:03.49133364 +0000 UTC m=+0.394907510,LastTimestamp:2026-07-17 18:33:03.49133364 +0000 UTC m=+0.394907510,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-220.ec2.internal,}" Jul 17 18:33:03.515734 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.515644 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-220.ec2.internal.18c32762fe0c8017 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-220.ec2.internal,UID:ip-10-0-132-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-220.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-220.ec2.internal,},FirstTimestamp:2026-07-17 18:33:03.491358743 +0000 UTC m=+0.394932611,LastTimestamp:2026-07-17 18:33:03.491358743 +0000 UTC m=+0.394932611,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-220.ec2.internal,}" Jul 17 18:33:03.519230 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.519147 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-220.ec2.internal.18c32762fe0cb4d4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-220.ec2.internal,UID:ip-10-0-132-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-220.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-220.ec2.internal,},FirstTimestamp:2026-07-17 18:33:03.491372244 +0000 UTC m=+0.394946112,LastTimestamp:2026-07-17 18:33:03.491372244 +0000 UTC m=+0.394946112,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-220.ec2.internal,}" Jul 17 18:33:03.529073 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.528980 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-220.ec2.internal.18c32762fe0c1e08\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-220.ec2.internal.18c32762fe0c1e08 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-220.ec2.internal,UID:ip-10-0-132-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-220.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-220.ec2.internal,},FirstTimestamp:2026-07-17 18:33:03.49133364 +0000 UTC m=+0.394907510,LastTimestamp:2026-07-17 18:33:03.492802322 +0000 UTC m=+0.396376192,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-220.ec2.internal,}" Jul 17 18:33:03.533465 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.533427 2563 manager.go:337] "Starting Device Plugin manager" Jul 17 18:33:03.533567 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.533503 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 18:33:03.533567 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.533514 2563 server.go:87] "Starting device plugin registration server" Jul 17 18:33:03.533844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.533829 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 18:33:03.533911 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.533849 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 18:33:03.533959 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.533948 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 18:33:03.534091 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.534063 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 18:33:03.534091 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.534082 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 18:33:03.534683 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.534666 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 18:33:03.534773 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.534716 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:03.535181 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.535095 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-220.ec2.internal.18c32762fe0c8017\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-220.ec2.internal.18c32762fe0c8017 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-220.ec2.internal,UID:ip-10-0-132-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-220.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-220.ec2.internal,},FirstTimestamp:2026-07-17 18:33:03.491358743 +0000 UTC m=+0.394932611,LastTimestamp:2026-07-17 18:33:03.492825246 +0000 UTC m=+0.396399113,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-220.ec2.internal,}" Jul 17 18:33:03.542393 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.542301 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-220.ec2.internal.18c32762fe0cb4d4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-220.ec2.internal.18c32762fe0cb4d4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-220.ec2.internal,UID:ip-10-0-132-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-220.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-220.ec2.internal,},FirstTimestamp:2026-07-17 18:33:03.491372244 +0000 UTC m=+0.394946112,LastTimestamp:2026-07-17 18:33:03.492835521 +0000 UTC m=+0.396409388,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-220.ec2.internal,}" Jul 17 18:33:03.548375 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.548269 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-220.ec2.internal.18c32763014dab12 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-220.ec2.internal,UID:ip-10-0-132-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-132-220.ec2.internal,},FirstTimestamp:2026-07-17 18:33:03.545961234 +0000 UTC m=+0.449535092,LastTimestamp:2026-07-17 18:33:03.545961234 +0000 UTC m=+0.449535092,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-220.ec2.internal,}" Jul 17 18:33:03.581204 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.581171 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6v8n" Jul 17 18:33:03.598120 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.598087 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6v8n" Jul 17 18:33:03.615839 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.615814 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 18:33:03.615935 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.615862 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 18:33:03.615935 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.615886 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 18:33:03.615935 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.615922 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 18:33:03.630775 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.630736 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:33:03.633967 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.633919 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:33:03.634894 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.634873 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:33:03.635014 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.634904 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:33:03.635014 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.634913 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:33:03.635014 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.634935 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.647369 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.647339 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.647369 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.647368 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-220.ec2.internal\": node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:03.694462 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.694407 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:03.716314 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.716278 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal"] Jul 17 18:33:03.716492 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.716405 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:33:03.717404 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.717378 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:33:03.717537 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.717412 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:33:03.717537 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.717424 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:33:03.718682 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.718670 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:33:03.718846 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.718832 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.718883 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.718864 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:33:03.719469 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.719437 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:33:03.719554 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.719487 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:33:03.719554 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.719497 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:33:03.719554 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.719438 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:33:03.719695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.719561 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:33:03.719695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.719581 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:33:03.720791 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.720774 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.720890 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.720799 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:33:03.721526 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.721512 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:33:03.721619 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.721544 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:33:03.721619 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.721562 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:33:03.742362 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.742340 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-220.ec2.internal\" not found" node="ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.747066 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.747038 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-220.ec2.internal\" not found" node="ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.771818 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.771783 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91b21fd82dcac5e9146a1a46e63734b3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal\" (UID: \"91b21fd82dcac5e9146a1a46e63734b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.771818 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.771823 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91b21fd82dcac5e9146a1a46e63734b3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal\" (UID: \"91b21fd82dcac5e9146a1a46e63734b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.772009 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.771848 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ece1844396d1b4d185e7c0e925cd74d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-220.ec2.internal\" (UID: \"6ece1844396d1b4d185e7c0e925cd74d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.795075 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.795037 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:03.872686 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.872663 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91b21fd82dcac5e9146a1a46e63734b3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal\" (UID: \"91b21fd82dcac5e9146a1a46e63734b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.872799 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.872693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91b21fd82dcac5e9146a1a46e63734b3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal\" (UID: \"91b21fd82dcac5e9146a1a46e63734b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.872799 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.872709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ece1844396d1b4d185e7c0e925cd74d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-220.ec2.internal\" (UID: \"6ece1844396d1b4d185e7c0e925cd74d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.872799 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.872738 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ece1844396d1b4d185e7c0e925cd74d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-220.ec2.internal\" (UID: \"6ece1844396d1b4d185e7c0e925cd74d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.872799 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.872769 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/91b21fd82dcac5e9146a1a46e63734b3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal\" (UID: \"91b21fd82dcac5e9146a1a46e63734b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.872799 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:03.872787 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91b21fd82dcac5e9146a1a46e63734b3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal\" (UID: \"91b21fd82dcac5e9146a1a46e63734b3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" Jul 17 18:33:03.895863 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.895788 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:03.996832 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:03.996794 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:04.047429 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.047375 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" Jul 17 18:33:04.049652 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.049631 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal" Jul 17 18:33:04.097639 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:04.097597 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:04.199394 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:04.199302 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:04.301404 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:04.301364 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:04.403372 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:04.403328 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:04.404550 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.404513 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 18:33:04.404708 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.404687 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 18:33:04.504523 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:04.504405 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:04.526691 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.526655 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 18:33:04.527706 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.527686 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientMemory" Jul 17 18:33:04.527823 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.527722 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 18:33:04.527823 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.527740 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeHasSufficientPID" Jul 17 18:33:04.602649 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.602590 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 18:28:04 +0000 UTC" deadline="2028-03-15 16:24:17.459728299 +0000 UTC" Jul 17 18:33:04.602649 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.602636 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14565h51m12.857096163s" Jul 17 18:33:04.604921 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:04.604902 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-220.ec2.internal\" not found" Jul 17 18:33:04.650400 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.650371 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:33:04.669797 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.669760 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" Jul 17 18:33:04.677771 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.677740 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:33:04.693465 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.693431 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 18:33:04.694219 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.694203 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal" Jul 17 18:33:04.708737 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.708707 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 18:33:04.749884 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:04.749849 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b21fd82dcac5e9146a1a46e63734b3.slice/crio-044628f3638c1b1a3eb5aa4a89711b6a3f13c3dd8b75550725dca7a0bcbf3027 WatchSource:0}: Error finding container 044628f3638c1b1a3eb5aa4a89711b6a3f13c3dd8b75550725dca7a0bcbf3027: Status 404 returned error can't find the container with id 044628f3638c1b1a3eb5aa4a89711b6a3f13c3dd8b75550725dca7a0bcbf3027 Jul 17 18:33:04.750432 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:04.750409 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ece1844396d1b4d185e7c0e925cd74d.slice/crio-e3dc2455c2106cea68b5e8c837e2e601a5d05adf3f9a231cda19018e61d0d19c WatchSource:0}: Error finding container e3dc2455c2106cea68b5e8c837e2e601a5d05adf3f9a231cda19018e61d0d19c: Status 404 returned error can't find the container with id e3dc2455c2106cea68b5e8c837e2e601a5d05adf3f9a231cda19018e61d0d19c Jul 17 18:33:04.758728 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:04.758628 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:33:05.046018 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.045919 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:33:05.380218 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.380178 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 18:33:05.451210 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.451170 2563 apiserver.go:52] "Watching apiserver" Jul 17 18:33:05.460582 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.460547 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 18:33:05.461092 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.461064 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-5kmz6","openshift-ovn-kubernetes/ovnkube-node-8jqr2","kube-system/global-pull-secret-syncer-kccpg","kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal","openshift-dns/node-resolver-tkrs6","openshift-image-registry/node-ca-qnnsp","openshift-multus/multus-4p2c6","openshift-multus/network-metrics-daemon-6jqfb","openshift-network-diagnostics/network-check-target-z7rdv","openshift-network-operator/iptables-alerter-jxdss","kube-system/konnectivity-agent-rkn6w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt","openshift-cluster-node-tuning-operator/tuned-4j7gh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal"] Jul 17 18:33:05.463000 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.462976 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.464971 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.464948 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.466421 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.466397 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:05.466543 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.466503 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:05.467493 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.467468 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dlzlv\"" Jul 17 18:33:05.467618 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.467501 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 18:33:05.467838 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.467814 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:05.467971 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.467911 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:05.468475 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.468448 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:33:05.468475 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.468518 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 18:33:05.468866 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.468842 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 18:33:05.468964 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.468875 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m8m4t\"" Jul 17 18:33:05.468964 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.468917 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:33:05.469386 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.469364 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.470956 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.470768 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 18:33:05.470956 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.470837 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.472577 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.472557 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:05.472675 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.472647 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:05.473922 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.473898 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.475966 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.475945 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.477934 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.477912 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.479064 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.479045 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 18:33:05.479326 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.479305 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 18:33:05.480230 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.479515 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 18:33:05.480230 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.479702 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 18:33:05.480230 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.480219 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:05.480437 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.480387 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 18:33:05.481521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.480594 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 18:33:05.481521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.480697 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 18:33:05.481521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.480839 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2znwt\"" Jul 17 18:33:05.481521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.480968 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 18:33:05.481521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.481041 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 18:33:05.481521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.481091 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k858s\"" Jul 17 18:33:05.481521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.481209 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 18:33:05.481521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.481237 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 18:33:05.481521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.481301 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 18:33:05.481521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.481477 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 18:33:05.482248 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.481734 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 18:33:05.482248 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482177 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-89dk4\"" Jul 17 18:33:05.482248 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482222 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12b80772-f189-48ae-ba3a-44f08da802b6-iptables-alerter-script\") pod \"iptables-alerter-jxdss\" (UID: \"12b80772-f189-48ae-ba3a-44f08da802b6\") " pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.482360 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482253 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-run\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482360 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482281 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-var-lib-kubelet\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482360 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482332 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f163c09b-fe4d-48de-9c56-7c8bbe212bc1-tmp-dir\") pod \"node-resolver-tkrs6\" (UID: \"f163c09b-fe4d-48de-9c56-7c8bbe212bc1\") " pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.482489 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482367 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f74a126-e2b2-4930-91f5-d8fab73ba4af-kubelet-config\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:05.482489 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482412 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f74a126-e2b2-4930-91f5-d8fab73ba4af-dbus\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:05.482489 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482441 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f163c09b-fe4d-48de-9c56-7c8bbe212bc1-hosts-file\") pod \"node-resolver-tkrs6\" (UID: \"f163c09b-fe4d-48de-9c56-7c8bbe212bc1\") " pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.482616 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482488 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sb9w\" (UniqueName: \"kubernetes.io/projected/f163c09b-fe4d-48de-9c56-7c8bbe212bc1-kube-api-access-8sb9w\") pod \"node-resolver-tkrs6\" (UID: \"f163c09b-fe4d-48de-9c56-7c8bbe212bc1\") " pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.482616 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482514 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:05.482616 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482551 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-sysctl-conf\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482616 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-lib-modules\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482616 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482597 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-host\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482770 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482621 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-tuned\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482770 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482647 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:05.482770 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482688 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-modprobe-d\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482770 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482711 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-sysconfig\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482770 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-kubernetes\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482770 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482760 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c-serviceca\") pod \"node-ca-qnnsp\" (UID: \"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c\") " pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.482966 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482786 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-systemd\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482966 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482809 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-tmp\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.482966 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482834 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js5qv\" (UniqueName: \"kubernetes.io/projected/7e6936f2-9870-44cc-abd2-99ae76f575ff-kube-api-access-js5qv\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:05.482966 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.482873 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-sysctl-d\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.483189 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.483107 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dcf4\" (UniqueName: \"kubernetes.io/projected/d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c-kube-api-access-9dcf4\") pod \"node-ca-qnnsp\" (UID: \"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c\") " pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.483189 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.483124 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hxr6b\"" Jul 17 18:33:05.483189 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.483137 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12b80772-f189-48ae-ba3a-44f08da802b6-host-slash\") pod \"iptables-alerter-jxdss\" (UID: \"12b80772-f189-48ae-ba3a-44f08da802b6\") " pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.483189 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.483163 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-sys\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.483325 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.483189 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c-host\") pod \"node-ca-qnnsp\" (UID: \"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c\") " pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.483325 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.483212 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdrkc\" (UniqueName: \"kubernetes.io/projected/12b80772-f189-48ae-ba3a-44f08da802b6-kube-api-access-gdrkc\") pod \"iptables-alerter-jxdss\" (UID: \"12b80772-f189-48ae-ba3a-44f08da802b6\") " pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.483325 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.483236 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5nf\" (UniqueName: \"kubernetes.io/projected/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-kube-api-access-6x5nf\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.483325 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.483312 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.489079 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.489052 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 18:33:05.489427 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.489401 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 18:33:05.489584 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.489557 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 18:33:05.489964 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.489944 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 18:33:05.490039 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.489994 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 18:33:05.490275 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.490231 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 18:33:05.490275 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.490252 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 18:33:05.490425 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.490252 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p2qql\"" Jul 17 18:33:05.490425 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.490240 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mhxkn\"" Jul 17 18:33:05.495369 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.494988 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 18:33:05.495369 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.495168 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-plbvz\"" Jul 17 18:33:05.513305 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.513275 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 18:33:05.572232 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.572198 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 18:33:05.583805 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.583767 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-run-multus-certs\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.584006 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.583816 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-node-log\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.584006 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.583842 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/76551af8-56c3-4d22-923f-4df4c58e1ab6-ovnkube-config\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.584006 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.583866 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-registration-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.584006 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.583892 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-etc-selinux\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.584006 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.583924 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c-serviceca\") pod \"node-ca-qnnsp\" (UID: \"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c\") " pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.584006 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.583948 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-systemd\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.584006 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.583986 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-cni-binary-copy\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584011 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/76551af8-56c3-4d22-923f-4df4c58e1ab6-env-overrides\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584036 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm7tk\" (UniqueName: \"kubernetes.io/projected/f3217f73-aa32-4cf7-b9ea-771cab429a4f-kube-api-access-wm7tk\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584069 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dcf4\" (UniqueName: \"kubernetes.io/projected/d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c-kube-api-access-9dcf4\") pod \"node-ca-qnnsp\" (UID: \"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c\") " pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584097 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-socket-dir-parent\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584124 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-conf-dir\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584150 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scdsc\" (UniqueName: \"kubernetes.io/projected/76551af8-56c3-4d22-923f-4df4c58e1ab6-kube-api-access-scdsc\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584175 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-device-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584203 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c-host\") pod \"node-ca-qnnsp\" (UID: \"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c\") " pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5nf\" (UniqueName: \"kubernetes.io/projected/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-kube-api-access-6x5nf\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584253 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-etc-kubernetes\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584280 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584305 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584331 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-socket-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.584362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584359 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cmqf\" (UniqueName: \"kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf\") pod \"network-check-target-z7rdv\" (UID: \"30bd0145-a321-42a2-8607-83268d918215\") " pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584386 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-run-openvswitch\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584412 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-hostroot\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584443 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-slash\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584469 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/76551af8-56c3-4d22-923f-4df4c58e1ab6-ovnkube-script-lib\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f74a126-e2b2-4930-91f5-d8fab73ba4af-dbus\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584562 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sb9w\" (UniqueName: \"kubernetes.io/projected/f163c09b-fe4d-48de-9c56-7c8bbe212bc1-kube-api-access-8sb9w\") pod \"node-resolver-tkrs6\" (UID: \"f163c09b-fe4d-48de-9c56-7c8bbe212bc1\") " pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584589 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-var-lib-cni-multus\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584629 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbxv\" (UniqueName: \"kubernetes.io/projected/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-kube-api-access-5sbxv\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584623 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c-serviceca\") pod \"node-ca-qnnsp\" (UID: \"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c\") " pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584673 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3217f73-aa32-4cf7-b9ea-771cab429a4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584716 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-run-netns\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584746 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-cni-bin\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584768 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69e728c7-854c-4ff2-a3e6-648a15b9c47f-agent-certs\") pod \"konnectivity-agent-rkn6w\" (UID: \"69e728c7-854c-4ff2-a3e6-648a15b9c47f\") " pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584798 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69e728c7-854c-4ff2-a3e6-648a15b9c47f-konnectivity-ca\") pod \"konnectivity-agent-rkn6w\" (UID: \"69e728c7-854c-4ff2-a3e6-648a15b9c47f\") " pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584889 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584936 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-tuned\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.585057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584966 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-system-cni-dir\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.584994 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-etc-openvswitch\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585021 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-system-cni-dir\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-modprobe-d\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585104 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-sysconfig\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-kubernetes\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585186 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-run-netns\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585202 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-run-systemd\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585203 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c-host\") pod \"node-ca-qnnsp\" (UID: \"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c\") " pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585229 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/76551af8-56c3-4d22-923f-4df4c58e1ab6-ovn-node-metrics-cert\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585253 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-tmp\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585273 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js5qv\" (UniqueName: \"kubernetes.io/projected/7e6936f2-9870-44cc-abd2-99ae76f575ff-kube-api-access-js5qv\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-cni-dir\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585314 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-os-release\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585479 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-os-release\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585516 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-sysctl-d\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.585977 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585570 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12b80772-f189-48ae-ba3a-44f08da802b6-host-slash\") pod \"iptables-alerter-jxdss\" (UID: \"12b80772-f189-48ae-ba3a-44f08da802b6\") " pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585608 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-sys\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585637 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-var-lib-openvswitch\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585665 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-log-socket\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585693 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585722 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-cnibin\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585750 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3217f73-aa32-4cf7-b9ea-771cab429a4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585783 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdrkc\" (UniqueName: \"kubernetes.io/projected/12b80772-f189-48ae-ba3a-44f08da802b6-kube-api-access-gdrkc\") pod \"iptables-alerter-jxdss\" (UID: \"12b80772-f189-48ae-ba3a-44f08da802b6\") " pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585839 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-run-k8s-cni-cncf-io\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585872 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-kubelet\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585901 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-cni-netd\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585928 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-sys-fs\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.585970 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhfhf\" (UniqueName: \"kubernetes.io/projected/149ffd16-ce3e-494d-a8db-4cbceb232d6e-kube-api-access-hhfhf\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12b80772-f189-48ae-ba3a-44f08da802b6-iptables-alerter-script\") pod \"iptables-alerter-jxdss\" (UID: \"12b80772-f189-48ae-ba3a-44f08da802b6\") " pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-run\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586065 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-var-lib-kubelet\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.586844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f163c09b-fe4d-48de-9c56-7c8bbe212bc1-tmp-dir\") pod \"node-resolver-tkrs6\" (UID: \"f163c09b-fe4d-48de-9c56-7c8bbe212bc1\") " pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586129 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-var-lib-cni-bin\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586159 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f74a126-e2b2-4930-91f5-d8fab73ba4af-kubelet-config\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f163c09b-fe4d-48de-9c56-7c8bbe212bc1-hosts-file\") pod \"node-resolver-tkrs6\" (UID: \"f163c09b-fe4d-48de-9c56-7c8bbe212bc1\") " pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586216 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-cnibin\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586244 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-daemon-config\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586288 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-run-ovn\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586350 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-var-lib-kubelet\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586377 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-systemd-units\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586410 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3217f73-aa32-4cf7-b9ea-771cab429a4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586448 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-sysctl-conf\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586691 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-lib-modules\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586726 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-host\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586832 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-host\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.586982 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-sysctl-conf\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.587033 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-systemd\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.587632 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.587391 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:05.588419 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.587482 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret podName:0f74a126-e2b2-4930-91f5-d8fab73ba4af nodeName:}" failed. No retries permitted until 2026-07-17 18:33:06.087449106 +0000 UTC m=+2.991022980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret") pod "global-pull-secret-syncer-kccpg" (UID: "0f74a126-e2b2-4930-91f5-d8fab73ba4af") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:05.588419 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.587970 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 18:33:05.588419 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.588021 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:05.588419 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.588080 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs podName:7e6936f2-9870-44cc-abd2-99ae76f575ff nodeName:}" failed. No retries permitted until 2026-07-17 18:33:06.088063938 +0000 UTC m=+2.991637799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs") pod "network-metrics-daemon-6jqfb" (UID: "7e6936f2-9870-44cc-abd2-99ae76f575ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:05.588419 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.588172 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-modprobe-d\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.588419 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.588222 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-sysconfig\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.588419 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.588262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-kubernetes\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.588853 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.588811 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f74a126-e2b2-4930-91f5-d8fab73ba4af-dbus\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:05.589139 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.589117 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f163c09b-fe4d-48de-9c56-7c8bbe212bc1-tmp-dir\") pod \"node-resolver-tkrs6\" (UID: \"f163c09b-fe4d-48de-9c56-7c8bbe212bc1\") " pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.590075 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.590042 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/12b80772-f189-48ae-ba3a-44f08da802b6-iptables-alerter-script\") pod \"iptables-alerter-jxdss\" (UID: \"12b80772-f189-48ae-ba3a-44f08da802b6\") " pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.590177 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.590116 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-run\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.590245 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.590172 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-var-lib-kubelet\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.590306 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.590278 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-sysctl-d\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.590362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.590346 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f74a126-e2b2-4930-91f5-d8fab73ba4af-kubelet-config\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:05.590456 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.590392 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f163c09b-fe4d-48de-9c56-7c8bbe212bc1-hosts-file\") pod \"node-resolver-tkrs6\" (UID: \"f163c09b-fe4d-48de-9c56-7c8bbe212bc1\") " pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.590456 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.590451 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-sys\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.590456 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.590484 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12b80772-f189-48ae-ba3a-44f08da802b6-host-slash\") pod \"iptables-alerter-jxdss\" (UID: \"12b80772-f189-48ae-ba3a-44f08da802b6\") " pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.590668 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.590636 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-lib-modules\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.592921 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.592858 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-tmp\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.593339 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.593308 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-etc-tuned\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.602879 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.602773 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdrkc\" (UniqueName: \"kubernetes.io/projected/12b80772-f189-48ae-ba3a-44f08da802b6-kube-api-access-gdrkc\") pod \"iptables-alerter-jxdss\" (UID: \"12b80772-f189-48ae-ba3a-44f08da802b6\") " pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.602879 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.602822 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5nf\" (UniqueName: \"kubernetes.io/projected/832f2fb1-bc13-4cd9-b44b-f7421b8d4fab-kube-api-access-6x5nf\") pod \"tuned-4j7gh\" (UID: \"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab\") " pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.604895 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.604862 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dcf4\" (UniqueName: \"kubernetes.io/projected/d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c-kube-api-access-9dcf4\") pod \"node-ca-qnnsp\" (UID: \"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c\") " pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.606464 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.606423 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kdvth" Jul 17 18:33:05.615890 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.615812 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js5qv\" (UniqueName: \"kubernetes.io/projected/7e6936f2-9870-44cc-abd2-99ae76f575ff-kube-api-access-js5qv\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:05.616867 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.616836 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sb9w\" (UniqueName: \"kubernetes.io/projected/f163c09b-fe4d-48de-9c56-7c8bbe212bc1-kube-api-access-8sb9w\") pod \"node-resolver-tkrs6\" (UID: \"f163c09b-fe4d-48de-9c56-7c8bbe212bc1\") " pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.617745 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.617716 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kdvth" Jul 17 18:33:05.622301 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.622229 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" event={"ID":"91b21fd82dcac5e9146a1a46e63734b3","Type":"ContainerStarted","Data":"044628f3638c1b1a3eb5aa4a89711b6a3f13c3dd8b75550725dca7a0bcbf3027"} Jul 17 18:33:05.624196 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.624156 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal" event={"ID":"6ece1844396d1b4d185e7c0e925cd74d","Type":"ContainerStarted","Data":"e3dc2455c2106cea68b5e8c837e2e601a5d05adf3f9a231cda19018e61d0d19c"} Jul 17 18:33:05.687569 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687524 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/76551af8-56c3-4d22-923f-4df4c58e1ab6-ovnkube-config\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.687764 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687580 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-registration-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.687764 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-etc-selinux\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.687764 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687630 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-cni-binary-copy\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.687764 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687668 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/76551af8-56c3-4d22-923f-4df4c58e1ab6-env-overrides\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.687764 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7tk\" (UniqueName: \"kubernetes.io/projected/f3217f73-aa32-4cf7-b9ea-771cab429a4f-kube-api-access-wm7tk\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.687764 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687722 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-socket-dir-parent\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.687764 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687746 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-conf-dir\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687772 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scdsc\" (UniqueName: \"kubernetes.io/projected/76551af8-56c3-4d22-923f-4df4c58e1ab6-kube-api-access-scdsc\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-device-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-etc-kubernetes\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687891 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687951 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-socket-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.687976 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmqf\" (UniqueName: \"kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf\") pod \"network-check-target-z7rdv\" (UID: \"30bd0145-a321-42a2-8607-83268d918215\") " pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-run-openvswitch\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688025 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-hostroot\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-slash\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688075 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/76551af8-56c3-4d22-923f-4df4c58e1ab6-ovnkube-script-lib\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688103 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-var-lib-cni-multus\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688128 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbxv\" (UniqueName: \"kubernetes.io/projected/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-kube-api-access-5sbxv\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688155 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3217f73-aa32-4cf7-b9ea-771cab429a4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688180 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-run-netns\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-cni-bin\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688208 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/76551af8-56c3-4d22-923f-4df4c58e1ab6-ovnkube-config\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69e728c7-854c-4ff2-a3e6-648a15b9c47f-agent-certs\") pod \"konnectivity-agent-rkn6w\" (UID: \"69e728c7-854c-4ff2-a3e6-648a15b9c47f\") " pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69e728c7-854c-4ff2-a3e6-648a15b9c47f-konnectivity-ca\") pod \"konnectivity-agent-rkn6w\" (UID: \"69e728c7-854c-4ff2-a3e6-648a15b9c47f\") " pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688296 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-system-cni-dir\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688320 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-etc-openvswitch\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688347 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-system-cni-dir\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688379 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688389 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-run-netns\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688423 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-run-systemd\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688438 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-registration-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688449 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/76551af8-56c3-4d22-923f-4df4c58e1ab6-ovn-node-metrics-cert\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.688482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688474 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-cni-dir\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688495 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-os-release\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688516 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-os-release\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688568 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-var-lib-openvswitch\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688570 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-etc-selinux\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688595 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-log-socket\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688652 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-cnibin\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3217f73-aa32-4cf7-b9ea-771cab429a4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-run-k8s-cni-cncf-io\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688738 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-kubelet\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688763 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-cni-netd\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688788 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-sys-fs\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688816 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhfhf\" (UniqueName: \"kubernetes.io/projected/149ffd16-ce3e-494d-a8db-4cbceb232d6e-kube-api-access-hhfhf\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688852 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-var-lib-cni-bin\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688881 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-cnibin\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688907 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-daemon-config\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688935 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-run-ovn\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688964 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688970 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/76551af8-56c3-4d22-923f-4df4c58e1ab6-env-overrides\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.688990 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-var-lib-kubelet\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689017 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-systemd-units\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3217f73-aa32-4cf7-b9ea-771cab429a4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689075 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-run-multus-certs\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-node-log\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689228 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-node-log\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689343 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-socket-dir-parent\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689358 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-socket-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689393 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-conf-dir\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689576 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-run-openvswitch\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689648 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-hostroot\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689700 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-slash\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689730 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-device-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689780 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-etc-kubernetes\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689825 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-run-ovn-kubernetes\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.689864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.689866 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-var-lib-cni-multus\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.690425 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.690315 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/76551af8-56c3-4d22-923f-4df4c58e1ab6-ovnkube-script-lib\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.690677 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.690647 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3217f73-aa32-4cf7-b9ea-771cab429a4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.690797 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.690716 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-run-netns\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.690797 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.690762 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-cni-bin\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.690889 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.690858 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-cni-binary-copy\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.690926 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.690914 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.690993 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.690964 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-cnibin\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.691190 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691155 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-cnibin\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.691288 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691222 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-cni-netd\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.691288 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691264 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-var-lib-kubelet\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.691288 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691252 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-sys-fs\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.691453 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691286 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-host-kubelet\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.691453 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691327 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-run-k8s-cni-cncf-io\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.691453 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691370 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-system-cni-dir\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.691638 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691487 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-etc-openvswitch\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.691638 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691519 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-system-cni-dir\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.691638 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691555 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-run-netns\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.691785 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691645 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-systemd-units\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.691785 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691695 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f3217f73-aa32-4cf7-b9ea-771cab429a4f-os-release\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.691785 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691708 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/149ffd16-ce3e-494d-a8db-4cbceb232d6e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.691785 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691703 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-var-lib-cni-bin\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.691785 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691750 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-log-socket\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.691785 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691754 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-run-systemd\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.691785 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691784 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-host-run-multus-certs\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.692215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691791 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-run-ovn\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.692215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691876 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3217f73-aa32-4cf7-b9ea-771cab429a4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.692215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691890 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-os-release\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.692215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.691964 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-cni-dir\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.692215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.692029 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/76551af8-56c3-4d22-923f-4df4c58e1ab6-var-lib-openvswitch\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.692215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.692086 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3217f73-aa32-4cf7-b9ea-771cab429a4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.692215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.692111 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-multus-daemon-config\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.692570 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.692349 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/69e728c7-854c-4ff2-a3e6-648a15b9c47f-konnectivity-ca\") pod \"konnectivity-agent-rkn6w\" (UID: \"69e728c7-854c-4ff2-a3e6-648a15b9c47f\") " pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:05.693992 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.693965 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/69e728c7-854c-4ff2-a3e6-648a15b9c47f-agent-certs\") pod \"konnectivity-agent-rkn6w\" (UID: \"69e728c7-854c-4ff2-a3e6-648a15b9c47f\") " pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:05.694747 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.694720 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/76551af8-56c3-4d22-923f-4df4c58e1ab6-ovn-node-metrics-cert\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.706962 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.706923 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:33:05.706962 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.706965 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:33:05.707219 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.706979 2563 projected.go:196] Error preparing data for projected volume kube-api-access-7cmqf for pod openshift-network-diagnostics/network-check-target-z7rdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:05.707219 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:05.707066 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf podName:30bd0145-a321-42a2-8607-83268d918215 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:06.20704511 +0000 UTC m=+3.110618985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmqf" (UniqueName: "kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf") pod "network-check-target-z7rdv" (UID: "30bd0145-a321-42a2-8607-83268d918215") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:05.710548 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.710507 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbxv\" (UniqueName: \"kubernetes.io/projected/031fe748-5d75-4fc9-a66a-de08e5d2b9d2-kube-api-access-5sbxv\") pod \"multus-4p2c6\" (UID: \"031fe748-5d75-4fc9-a66a-de08e5d2b9d2\") " pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.712023 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.711984 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7tk\" (UniqueName: \"kubernetes.io/projected/f3217f73-aa32-4cf7-b9ea-771cab429a4f-kube-api-access-wm7tk\") pod \"multus-additional-cni-plugins-5kmz6\" (UID: \"f3217f73-aa32-4cf7-b9ea-771cab429a4f\") " pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.714829 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.714794 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhfhf\" (UniqueName: \"kubernetes.io/projected/149ffd16-ce3e-494d-a8db-4cbceb232d6e-kube-api-access-hhfhf\") pod \"aws-ebs-csi-driver-node-8lplt\" (UID: \"149ffd16-ce3e-494d-a8db-4cbceb232d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.714973 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.714888 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scdsc\" (UniqueName: \"kubernetes.io/projected/76551af8-56c3-4d22-923f-4df4c58e1ab6-kube-api-access-scdsc\") pod \"ovnkube-node-8jqr2\" (UID: \"76551af8-56c3-4d22-923f-4df4c58e1ab6\") " pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.777277 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.777231 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jxdss" Jul 17 18:33:05.786059 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.786032 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" Jul 17 18:33:05.789565 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:05.789524 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b80772_f189_48ae_ba3a_44f08da802b6.slice/crio-bb2b72e7fdd8d5a8b62a9c2ca6387abd58d4893d044a39591f772c64ec3a88ac WatchSource:0}: Error finding container bb2b72e7fdd8d5a8b62a9c2ca6387abd58d4893d044a39591f772c64ec3a88ac: Status 404 returned error can't find the container with id bb2b72e7fdd8d5a8b62a9c2ca6387abd58d4893d044a39591f772c64ec3a88ac Jul 17 18:33:05.796297 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:05.796264 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832f2fb1_bc13_4cd9_b44b_f7421b8d4fab.slice/crio-e5890fd513ed36f77aeed22f6493c6cb5af82947b9434916359008a6e8c9e425 WatchSource:0}: Error finding container e5890fd513ed36f77aeed22f6493c6cb5af82947b9434916359008a6e8c9e425: Status 404 returned error can't find the container with id e5890fd513ed36f77aeed22f6493c6cb5af82947b9434916359008a6e8c9e425 Jul 17 18:33:05.797399 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.797318 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tkrs6" Jul 17 18:33:05.804939 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.804908 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qnnsp" Jul 17 18:33:05.806255 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:05.806216 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf163c09b_fe4d_48de_9c56_7c8bbe212bc1.slice/crio-669e2aaed68c9e907792dded6c6d0a7767c5a1d4935e3f003119c34f47052ce3 WatchSource:0}: Error finding container 669e2aaed68c9e907792dded6c6d0a7767c5a1d4935e3f003119c34f47052ce3: Status 404 returned error can't find the container with id 669e2aaed68c9e907792dded6c6d0a7767c5a1d4935e3f003119c34f47052ce3 Jul 17 18:33:05.812789 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.812764 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4p2c6" Jul 17 18:33:05.817535 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:05.817500 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ca6a7a_5112_46a9_99fc_7c54e3ca7c8c.slice/crio-2d65121e44aec3062135e00aebe28e77f68c88f86e4d8a06e85a1e65104312e9 WatchSource:0}: Error finding container 2d65121e44aec3062135e00aebe28e77f68c88f86e4d8a06e85a1e65104312e9: Status 404 returned error can't find the container with id 2d65121e44aec3062135e00aebe28e77f68c88f86e4d8a06e85a1e65104312e9 Jul 17 18:33:05.820697 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.820669 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:05.824589 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:05.824559 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031fe748_5d75_4fc9_a66a_de08e5d2b9d2.slice/crio-afce4ec6173128980efeee2ae18fa74e894fc7cd7792713f334b7963e2493be7 WatchSource:0}: Error finding container afce4ec6173128980efeee2ae18fa74e894fc7cd7792713f334b7963e2493be7: Status 404 returned error can't find the container with id afce4ec6173128980efeee2ae18fa74e894fc7cd7792713f334b7963e2493be7 Jul 17 18:33:05.829270 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.829171 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" Jul 17 18:33:05.831903 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:05.831835 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76551af8_56c3_4d22_923f_4df4c58e1ab6.slice/crio-e29b6d7f97d7a108570d715f95a30a8729c2de34eeadf6a2c8c90a97cd98e22e WatchSource:0}: Error finding container e29b6d7f97d7a108570d715f95a30a8729c2de34eeadf6a2c8c90a97cd98e22e: Status 404 returned error can't find the container with id e29b6d7f97d7a108570d715f95a30a8729c2de34eeadf6a2c8c90a97cd98e22e Jul 17 18:33:05.838810 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.838784 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:05.841758 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:05.841714 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3217f73_aa32_4cf7_b9ea_771cab429a4f.slice/crio-5220003d37972d39e29daacdeebe0544e7988f33c42c909e762d92f58ca1d1bb WatchSource:0}: Error finding container 5220003d37972d39e29daacdeebe0544e7988f33c42c909e762d92f58ca1d1bb: Status 404 returned error can't find the container with id 5220003d37972d39e29daacdeebe0544e7988f33c42c909e762d92f58ca1d1bb Jul 17 18:33:05.845789 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:05.845747 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" Jul 17 18:33:05.863009 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:05.862929 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e728c7_854c_4ff2_a3e6_648a15b9c47f.slice/crio-c16d90e98e44976598fd1f53c8228f219f2a4f611108f4dac800eef342a53511 WatchSource:0}: Error finding container c16d90e98e44976598fd1f53c8228f219f2a4f611108f4dac800eef342a53511: Status 404 returned error can't find the container with id c16d90e98e44976598fd1f53c8228f219f2a4f611108f4dac800eef342a53511 Jul 17 18:33:05.865257 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:05.865184 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149ffd16_ce3e_494d_a8db_4cbceb232d6e.slice/crio-d99441e5ab4a4bed5a238d8e98c946f9514effd335d6d1ccf96ff336bc045d2a WatchSource:0}: Error finding container d99441e5ab4a4bed5a238d8e98c946f9514effd335d6d1ccf96ff336bc045d2a: Status 404 returned error can't find the container with id d99441e5ab4a4bed5a238d8e98c946f9514effd335d6d1ccf96ff336bc045d2a Jul 17 18:33:06.092624 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.092584 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:06.092624 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.092632 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:06.092880 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:06.092770 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:06.092880 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:06.092844 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret podName:0f74a126-e2b2-4930-91f5-d8fab73ba4af nodeName:}" failed. No retries permitted until 2026-07-17 18:33:07.092819338 +0000 UTC m=+3.996393193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret") pod "global-pull-secret-syncer-kccpg" (UID: "0f74a126-e2b2-4930-91f5-d8fab73ba4af") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:06.093165 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:06.092771 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:06.093293 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:06.093193 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs podName:7e6936f2-9870-44cc-abd2-99ae76f575ff nodeName:}" failed. No retries permitted until 2026-07-17 18:33:07.093180041 +0000 UTC m=+3.996753897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs") pod "network-metrics-daemon-6jqfb" (UID: "7e6936f2-9870-44cc-abd2-99ae76f575ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:06.294589 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.294547 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmqf\" (UniqueName: \"kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf\") pod \"network-check-target-z7rdv\" (UID: \"30bd0145-a321-42a2-8607-83268d918215\") " pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:06.294807 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:06.294767 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:33:06.294807 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:06.294802 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:33:06.294985 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:06.294818 2563 projected.go:196] Error preparing data for projected volume kube-api-access-7cmqf for pod openshift-network-diagnostics/network-check-target-z7rdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:06.294985 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:06.294898 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf podName:30bd0145-a321-42a2-8607-83268d918215 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:07.294874258 +0000 UTC m=+4.198448115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmqf" (UniqueName: "kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf") pod "network-check-target-z7rdv" (UID: "30bd0145-a321-42a2-8607-83268d918215") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:06.616837 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.616745 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:06.617333 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:06.616869 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:06.620839 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.620790 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:28:06 +0000 UTC" deadline="2028-01-12 04:07:36.859347847 +0000 UTC" Jul 17 18:33:06.620839 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.620833 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13041h34m30.238521257s" Jul 17 18:33:06.627779 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.627742 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" event={"ID":"f3217f73-aa32-4cf7-b9ea-771cab429a4f","Type":"ContainerStarted","Data":"5220003d37972d39e29daacdeebe0544e7988f33c42c909e762d92f58ca1d1bb"} Jul 17 18:33:06.629125 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.629097 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" event={"ID":"76551af8-56c3-4d22-923f-4df4c58e1ab6","Type":"ContainerStarted","Data":"e29b6d7f97d7a108570d715f95a30a8729c2de34eeadf6a2c8c90a97cd98e22e"} Jul 17 18:33:06.630837 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.630811 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4p2c6" event={"ID":"031fe748-5d75-4fc9-a66a-de08e5d2b9d2","Type":"ContainerStarted","Data":"afce4ec6173128980efeee2ae18fa74e894fc7cd7792713f334b7963e2493be7"} Jul 17 18:33:06.631935 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.631912 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" event={"ID":"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab","Type":"ContainerStarted","Data":"e5890fd513ed36f77aeed22f6493c6cb5af82947b9434916359008a6e8c9e425"} Jul 17 18:33:06.633303 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.633277 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" event={"ID":"149ffd16-ce3e-494d-a8db-4cbceb232d6e","Type":"ContainerStarted","Data":"d99441e5ab4a4bed5a238d8e98c946f9514effd335d6d1ccf96ff336bc045d2a"} Jul 17 18:33:06.634615 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.634586 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qnnsp" event={"ID":"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c","Type":"ContainerStarted","Data":"2d65121e44aec3062135e00aebe28e77f68c88f86e4d8a06e85a1e65104312e9"} Jul 17 18:33:06.635958 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.635930 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tkrs6" event={"ID":"f163c09b-fe4d-48de-9c56-7c8bbe212bc1","Type":"ContainerStarted","Data":"669e2aaed68c9e907792dded6c6d0a7767c5a1d4935e3f003119c34f47052ce3"} Jul 17 18:33:06.637385 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.637353 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jxdss" event={"ID":"12b80772-f189-48ae-ba3a-44f08da802b6","Type":"ContainerStarted","Data":"bb2b72e7fdd8d5a8b62a9c2ca6387abd58d4893d044a39591f772c64ec3a88ac"} Jul 17 18:33:06.638482 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:06.638439 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rkn6w" event={"ID":"69e728c7-854c-4ff2-a3e6-648a15b9c47f","Type":"ContainerStarted","Data":"c16d90e98e44976598fd1f53c8228f219f2a4f611108f4dac800eef342a53511"} Jul 17 18:33:07.102004 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:07.101903 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:07.102004 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:07.101962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:07.102227 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:07.102128 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:07.102227 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:07.102198 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs podName:7e6936f2-9870-44cc-abd2-99ae76f575ff nodeName:}" failed. No retries permitted until 2026-07-17 18:33:09.102175332 +0000 UTC m=+6.005749189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs") pod "network-metrics-daemon-6jqfb" (UID: "7e6936f2-9870-44cc-abd2-99ae76f575ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:07.102743 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:07.102719 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:07.102879 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:07.102801 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret podName:0f74a126-e2b2-4930-91f5-d8fab73ba4af nodeName:}" failed. No retries permitted until 2026-07-17 18:33:09.102784227 +0000 UTC m=+6.006358098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret") pod "global-pull-secret-syncer-kccpg" (UID: "0f74a126-e2b2-4930-91f5-d8fab73ba4af") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:07.303872 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:07.303828 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmqf\" (UniqueName: \"kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf\") pod \"network-check-target-z7rdv\" (UID: \"30bd0145-a321-42a2-8607-83268d918215\") " pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:07.304115 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:07.304093 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:33:07.304237 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:07.304126 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:33:07.304237 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:07.304144 2563 projected.go:196] Error preparing data for projected volume kube-api-access-7cmqf for pod openshift-network-diagnostics/network-check-target-z7rdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:07.304237 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:07.304223 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf podName:30bd0145-a321-42a2-8607-83268d918215 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:09.304202369 +0000 UTC m=+6.207776230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmqf" (UniqueName: "kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf") pod "network-check-target-z7rdv" (UID: "30bd0145-a321-42a2-8607-83268d918215") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:07.617747 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:07.617658 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:07.618335 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:07.617824 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:07.625242 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:07.621549 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:07.625242 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:07.621644 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 18:28:06 +0000 UTC" deadline="2028-03-31 07:22:39.32229626 +0000 UTC" Jul 17 18:33:07.625242 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:07.621678 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14940h49m31.70062247s" Jul 17 18:33:07.625242 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:07.621743 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:08.617189 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:08.616792 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:08.617373 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:08.617316 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:08.669695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:08.669651 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" event={"ID":"91b21fd82dcac5e9146a1a46e63734b3","Type":"ContainerDied","Data":"7f84672b17c67f5c9661c0ad33f209ff7a882693356712251b358c50031b112d"} Jul 17 18:33:08.670893 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:08.670744 2563 generic.go:356] "Generic (PLEG): container finished" podID="91b21fd82dcac5e9146a1a46e63734b3" containerID="7f84672b17c67f5c9661c0ad33f209ff7a882693356712251b358c50031b112d" exitCode=0 Jul 17 18:33:08.682960 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:08.682383 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal" event={"ID":"6ece1844396d1b4d185e7c0e925cd74d","Type":"ContainerStarted","Data":"fd45ae6746cf177080695d4d2923f1ef708b6b421ffb36371cf03fe933322e4e"} Jul 17 18:33:09.121748 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:09.120770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:09.121748 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:09.120829 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:09.121748 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:09.120986 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:09.121748 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:09.121056 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs podName:7e6936f2-9870-44cc-abd2-99ae76f575ff nodeName:}" failed. No retries permitted until 2026-07-17 18:33:13.121033956 +0000 UTC m=+10.024607860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs") pod "network-metrics-daemon-6jqfb" (UID: "7e6936f2-9870-44cc-abd2-99ae76f575ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:09.121748 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:09.121474 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:09.121748 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:09.121543 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret podName:0f74a126-e2b2-4930-91f5-d8fab73ba4af nodeName:}" failed. No retries permitted until 2026-07-17 18:33:13.121509663 +0000 UTC m=+10.025083534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret") pod "global-pull-secret-syncer-kccpg" (UID: "0f74a126-e2b2-4930-91f5-d8fab73ba4af") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:09.323127 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:09.323080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmqf\" (UniqueName: \"kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf\") pod \"network-check-target-z7rdv\" (UID: \"30bd0145-a321-42a2-8607-83268d918215\") " pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:09.323331 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:09.323309 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:33:09.323418 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:09.323345 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:33:09.323418 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:09.323359 2563 projected.go:196] Error preparing data for projected volume kube-api-access-7cmqf for pod openshift-network-diagnostics/network-check-target-z7rdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:09.323584 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:09.323423 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf podName:30bd0145-a321-42a2-8607-83268d918215 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:13.32340364 +0000 UTC m=+10.226977515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmqf" (UniqueName: "kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf") pod "network-check-target-z7rdv" (UID: "30bd0145-a321-42a2-8607-83268d918215") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:09.616200 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:09.616102 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:09.616403 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:09.616238 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:09.618645 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:09.618604 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:09.618830 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:09.618747 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:09.707356 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:09.707243 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" event={"ID":"91b21fd82dcac5e9146a1a46e63734b3","Type":"ContainerStarted","Data":"6917b29f1d14aa1da97fcb94be744bbc1158e84bf0a413ab5fe5723048d6558d"} Jul 17 18:33:09.739488 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:09.739410 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-220.ec2.internal" podStartSLOduration=4.73937585 podStartE2EDuration="4.73937585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:08.732471904 +0000 UTC m=+5.636045786" watchObservedRunningTime="2026-07-17 18:33:09.73937585 +0000 UTC m=+6.642949710" Jul 17 18:33:09.740453 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:09.740401 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-220.ec2.internal" podStartSLOduration=4.740383619 podStartE2EDuration="4.740383619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:09.740051589 +0000 UTC m=+6.643625470" watchObservedRunningTime="2026-07-17 18:33:09.740383619 +0000 UTC m=+6.643957500" Jul 17 18:33:10.616647 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:10.616608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:10.616841 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:10.616739 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:11.616217 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:11.616172 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:11.616736 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:11.616324 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:11.616736 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:11.616692 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:11.616851 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:11.616832 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:12.616653 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:12.616618 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:12.617066 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:12.616744 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:13.158287 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:13.158031 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:13.158287 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:13.158088 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:13.158287 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:13.158209 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:13.158287 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:13.158228 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:13.158287 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:13.158281 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret podName:0f74a126-e2b2-4930-91f5-d8fab73ba4af nodeName:}" failed. No retries permitted until 2026-07-17 18:33:21.158264283 +0000 UTC m=+18.061838143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret") pod "global-pull-secret-syncer-kccpg" (UID: "0f74a126-e2b2-4930-91f5-d8fab73ba4af") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:13.158287 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:13.158302 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs podName:7e6936f2-9870-44cc-abd2-99ae76f575ff nodeName:}" failed. No retries permitted until 2026-07-17 18:33:21.158292683 +0000 UTC m=+18.061866540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs") pod "network-metrics-daemon-6jqfb" (UID: "7e6936f2-9870-44cc-abd2-99ae76f575ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:13.360112 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:13.359412 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmqf\" (UniqueName: \"kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf\") pod \"network-check-target-z7rdv\" (UID: \"30bd0145-a321-42a2-8607-83268d918215\") " pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:13.360112 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:13.359646 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:33:13.360112 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:13.359672 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:33:13.360112 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:13.359685 2563 projected.go:196] Error preparing data for projected volume kube-api-access-7cmqf for pod openshift-network-diagnostics/network-check-target-z7rdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:13.360112 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:13.359744 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf podName:30bd0145-a321-42a2-8607-83268d918215 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:21.359729163 +0000 UTC m=+18.263303035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmqf" (UniqueName: "kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf") pod "network-check-target-z7rdv" (UID: "30bd0145-a321-42a2-8607-83268d918215") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:13.621029 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:13.620889 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:13.621506 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:13.621083 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:13.621586 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:13.621552 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:13.621719 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:13.621659 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:14.616642 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:14.616600 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:14.616826 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:14.616750 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:15.618692 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:15.618651 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:15.619168 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:15.618804 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:15.619168 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:15.619149 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:15.619269 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:15.619239 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:16.616434 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:16.616396 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:16.616599 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:16.616535 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:17.619124 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:17.619038 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:17.619543 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:17.619191 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:17.619637 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:17.619616 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:17.619716 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:17.619699 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:18.616210 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:18.616179 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:18.616408 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:18.616304 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:19.618604 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:19.618566 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:19.619100 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:19.618707 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:19.619100 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:19.619078 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:19.619240 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:19.619169 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:20.616707 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:20.616665 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:20.616919 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:20.616812 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:21.219916 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:21.219870 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:21.219916 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:21.219918 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:21.220394 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:21.220025 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:21.220394 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:21.220095 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs podName:7e6936f2-9870-44cc-abd2-99ae76f575ff nodeName:}" failed. No retries permitted until 2026-07-17 18:33:37.22008011 +0000 UTC m=+34.123653964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs") pod "network-metrics-daemon-6jqfb" (UID: "7e6936f2-9870-44cc-abd2-99ae76f575ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:21.220394 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:21.220025 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:21.220394 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:21.220183 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret podName:0f74a126-e2b2-4930-91f5-d8fab73ba4af nodeName:}" failed. No retries permitted until 2026-07-17 18:33:37.220171586 +0000 UTC m=+34.123745440 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret") pod "global-pull-secret-syncer-kccpg" (UID: "0f74a126-e2b2-4930-91f5-d8fab73ba4af") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:21.422121 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:21.421512 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmqf\" (UniqueName: \"kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf\") pod \"network-check-target-z7rdv\" (UID: \"30bd0145-a321-42a2-8607-83268d918215\") " pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:21.422309 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:21.422051 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:33:21.422309 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:21.422225 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:33:21.422309 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:21.422251 2563 projected.go:196] Error preparing data for projected volume kube-api-access-7cmqf for pod openshift-network-diagnostics/network-check-target-z7rdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:21.422502 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:21.422319 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf podName:30bd0145-a321-42a2-8607-83268d918215 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:37.422298438 +0000 UTC m=+34.325872291 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmqf" (UniqueName: "kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf") pod "network-check-target-z7rdv" (UID: "30bd0145-a321-42a2-8607-83268d918215") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:21.616699 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:21.616609 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:21.616857 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:21.616608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:21.616857 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:21.616761 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:21.616857 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:21.616824 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:22.616544 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:22.616499 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:22.617034 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:22.616633 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:23.616998 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:23.616964 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:23.617434 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:23.617078 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:23.617434 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:23.617161 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:23.617434 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:23.617271 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:24.616588 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:24.616552 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:24.616834 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:24.616669 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:25.616988 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:25.616937 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:25.617400 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:25.616937 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:25.617400 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:25.617103 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:25.617400 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:25.617120 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:26.616948 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.616614 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:26.617181 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:26.617040 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:26.747816 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.747767 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" event={"ID":"149ffd16-ce3e-494d-a8db-4cbceb232d6e","Type":"ContainerStarted","Data":"7a7484cb30380fd747a8dccdc52c419a4154ff68b8e1021e0131ced09c8946c1"} Jul 17 18:33:26.749373 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.749339 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qnnsp" event={"ID":"d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c","Type":"ContainerStarted","Data":"347ed398c061d9e759a8cb505d762570c20d5db512b225d6fee83a9dc836e62e"} Jul 17 18:33:26.750910 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.750879 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tkrs6" event={"ID":"f163c09b-fe4d-48de-9c56-7c8bbe212bc1","Type":"ContainerStarted","Data":"0bd2ae29ed2ec795ececc624e39fcf4205973d4439e6f8a42b7163e5d5dcf82d"} Jul 17 18:33:26.752408 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.752379 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rkn6w" event={"ID":"69e728c7-854c-4ff2-a3e6-648a15b9c47f","Type":"ContainerStarted","Data":"9581cfd0f33312aee2040713b8b76889751de8285e2683eb22e9dbf76e5525fb"} Jul 17 18:33:26.754117 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.754086 2563 generic.go:356] "Generic (PLEG): container finished" podID="f3217f73-aa32-4cf7-b9ea-771cab429a4f" containerID="60caacb33c65554f573ff6e5b684979f2e959e22e249050bf800d76493e85b80" exitCode=0 Jul 17 18:33:26.754247 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.754174 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" event={"ID":"f3217f73-aa32-4cf7-b9ea-771cab429a4f","Type":"ContainerDied","Data":"60caacb33c65554f573ff6e5b684979f2e959e22e249050bf800d76493e85b80"} Jul 17 18:33:26.757713 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.757652 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" event={"ID":"76551af8-56c3-4d22-923f-4df4c58e1ab6","Type":"ContainerStarted","Data":"e2e2e2a41161e69ea24e301eae97545033cfa46ac48cb706bfda752e46518c0b"} Jul 17 18:33:26.757778 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.757732 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" event={"ID":"76551af8-56c3-4d22-923f-4df4c58e1ab6","Type":"ContainerStarted","Data":"00acc23fe7789e92cca1616d2c76a1e90a7112f222b88668e5e16b0be7bce8be"} Jul 17 18:33:26.757848 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.757782 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" event={"ID":"76551af8-56c3-4d22-923f-4df4c58e1ab6","Type":"ContainerStarted","Data":"93128018bf62debe302e4cbf38e2e132f962b7f80decdc38b86c37aaa3d9d1bc"} Jul 17 18:33:26.757905 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.757845 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" event={"ID":"76551af8-56c3-4d22-923f-4df4c58e1ab6","Type":"ContainerStarted","Data":"00d004760c02c8c58df34ea07b1446461ee3e1927f81bb730acbd19d94475d3d"} Jul 17 18:33:26.757905 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.757892 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" event={"ID":"76551af8-56c3-4d22-923f-4df4c58e1ab6","Type":"ContainerStarted","Data":"08a4a5aa39a3a672a6e5fc241d5b57a3f233389b0b7af6acfc39fd7c723062f3"} Jul 17 18:33:26.757987 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.757938 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" event={"ID":"76551af8-56c3-4d22-923f-4df4c58e1ab6","Type":"ContainerStarted","Data":"adaa75ebefea7754579c4ebef330f81f11d4049115063353ed05f781433048b5"} Jul 17 18:33:26.759898 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.759437 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4p2c6" event={"ID":"031fe748-5d75-4fc9-a66a-de08e5d2b9d2","Type":"ContainerStarted","Data":"791143d63dec2eb19f0c80e0a8c8f7be612ec206353ab89289cf4518774fb505"} Jul 17 18:33:26.761107 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.761082 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" event={"ID":"832f2fb1-bc13-4cd9-b44b-f7421b8d4fab","Type":"ContainerStarted","Data":"ee6262b2c98b0194536c5d5ced9b5a93650e8e394e9dd959e906b64e5b740a5f"} Jul 17 18:33:26.831283 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.831217 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rkn6w" podStartSLOduration=3.044623226 podStartE2EDuration="22.831196041s" totalImagesPullingTime="19.786572815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:26.830918221 +0000 UTC m=+23.734492099" watchObservedRunningTime="2026-07-17 18:33:26.831196041 +0000 UTC m=+23.734769916" Jul 17 18:33:26.831985 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.831948 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qnnsp" podStartSLOduration=2.994083453 podStartE2EDuration="22.831935353s" totalImagesPullingTime="19.8378519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:26.782866859 +0000 UTC m=+23.686440735" watchObservedRunningTime="2026-07-17 18:33:26.831935353 +0000 UTC m=+23.735509229" Jul 17 18:33:26.930795 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.930740 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4p2c6" podStartSLOduration=3.034633958 podStartE2EDuration="22.930717945s" totalImagesPullingTime="19.896083987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:26.884885599 +0000 UTC m=+23.788459475" watchObservedRunningTime="2026-07-17 18:33:26.930717945 +0000 UTC m=+23.834291823" Jul 17 18:33:26.960999 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.960972 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 18:33:26.978302 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:26.978248 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4j7gh" podStartSLOduration=3.103594467 podStartE2EDuration="22.978231897s" totalImagesPullingTime="19.87463743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:26.978125353 +0000 UTC m=+23.881699229" watchObservedRunningTime="2026-07-17 18:33:26.978231897 +0000 UTC m=+23.881805774" Jul 17 18:33:27.007276 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:27.007225 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tkrs6" podStartSLOduration=3.163871819 podStartE2EDuration="23.007208504s" totalImagesPullingTime="19.843336685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:27.007108574 +0000 UTC m=+23.910682450" watchObservedRunningTime="2026-07-17 18:33:27.007208504 +0000 UTC m=+23.910782379" Jul 17 18:33:27.545844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:27.545528 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T18:33:26.960992506Z","UUID":"c7812768-8a14-4253-ace3-9636eeaa82c6","Handler":null,"Name":"","Endpoint":""} Jul 17 18:33:27.547376 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:27.547348 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 18:33:27.547560 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:27.547398 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 18:33:27.616991 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:27.616947 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:27.616991 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:27.616975 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:27.617237 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:27.617085 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:27.617237 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:27.617205 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:27.765987 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:27.765947 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" event={"ID":"149ffd16-ce3e-494d-a8db-4cbceb232d6e","Type":"ContainerStarted","Data":"f00a156b87ba4e59cfaff7068281ec3f54e44270062059a1eadd9f9e3ee63cb4"} Jul 17 18:33:27.767700 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:27.767640 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jxdss" event={"ID":"12b80772-f189-48ae-ba3a-44f08da802b6","Type":"ContainerStarted","Data":"1dd4e12a22b1282e43091248b8a5adcb11c376db7522f6ecc1c78648818c20c8"} Jul 17 18:33:27.811293 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:27.811170 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jxdss" podStartSLOduration=3.951592741 podStartE2EDuration="23.811147353s" totalImagesPullingTime="19.859554612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:27.811094488 +0000 UTC m=+24.714668377" watchObservedRunningTime="2026-07-17 18:33:27.811147353 +0000 UTC m=+24.714721231" Jul 17 18:33:28.616385 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:28.616295 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:28.616568 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:28.616405 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:28.773794 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:28.773754 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" event={"ID":"149ffd16-ce3e-494d-a8db-4cbceb232d6e","Type":"ContainerStarted","Data":"92ec3d39fdc48686eb0d2eaaaccf50195b001a3822ef9c682518737107be8e6f"} Jul 17 18:33:28.777614 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:28.777584 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" event={"ID":"76551af8-56c3-4d22-923f-4df4c58e1ab6","Type":"ContainerStarted","Data":"0aef2cd8f74f4a8d3f503e5bc4a751e72b62c388fbade0319df5a9b1df74c959"} Jul 17 18:33:28.796809 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:28.796748 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8lplt" podStartSLOduration=2.743284771 podStartE2EDuration="24.796733416s" totalImagesPullingTime="22.053448645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:28.796583558 +0000 UTC m=+25.700157434" watchObservedRunningTime="2026-07-17 18:33:28.796733416 +0000 UTC m=+25.700307291" Jul 17 18:33:29.616783 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:29.616734 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:29.617045 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:29.616747 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:29.617045 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:29.616890 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:29.617045 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:29.616954 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:29.991130 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:29.991093 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:29.991727 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:29.991707 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:30.616404 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:30.616367 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:30.616591 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:30.616522 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:31.616623 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:31.616298 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:31.617071 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:31.616299 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:31.617071 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:31.616721 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:31.617071 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:31.616851 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:32.616841 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:32.616800 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:32.617494 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:32.616911 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:32.788419 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:32.788383 2563 generic.go:356] "Generic (PLEG): container finished" podID="f3217f73-aa32-4cf7-b9ea-771cab429a4f" containerID="76ee1865e4fe56d47341ed6f7847a75b1ede7f4ba49d0e5a90c3bd9f15a83df4" exitCode=0 Jul 17 18:33:32.788618 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:32.788481 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" event={"ID":"f3217f73-aa32-4cf7-b9ea-771cab429a4f","Type":"ContainerDied","Data":"76ee1865e4fe56d47341ed6f7847a75b1ede7f4ba49d0e5a90c3bd9f15a83df4"} Jul 17 18:33:32.794208 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:32.794173 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" event={"ID":"76551af8-56c3-4d22-923f-4df4c58e1ab6","Type":"ContainerStarted","Data":"e2e27a6ebc003e6e46b953bd310796a976308165d2d1b514b3ee96dd4e419804"} Jul 17 18:33:32.794483 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:32.794465 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:32.794562 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:32.794547 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:32.794638 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:32.794615 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:32.811749 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:32.811719 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:32.812105 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:32.811901 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:33:32.861047 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:32.860983 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" podStartSLOduration=8.870714298 podStartE2EDuration="28.860963665s" totalImagesPullingTime="19.990249367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:32.860411856 +0000 UTC m=+29.763985731" watchObservedRunningTime="2026-07-17 18:33:32.860963665 +0000 UTC m=+29.764537556" Jul 17 18:33:33.618052 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:33.617853 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:33.618424 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:33.617922 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:33.618424 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:33.618128 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:33.618424 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:33.618206 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:34.596968 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.596934 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:34.597149 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.597136 2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 17 18:33:34.597628 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.597600 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rkn6w" Jul 17 18:33:34.612097 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.612063 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7rdv"] Jul 17 18:33:34.612245 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.612191 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:34.612315 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:34.612283 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:34.616363 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.616330 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:34.616506 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:34.616429 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:34.621141 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.620897 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jqfb"] Jul 17 18:33:34.621141 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.621050 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:34.621591 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:34.621167 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:34.621591 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.621561 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kccpg"] Jul 17 18:33:34.801284 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.801196 2563 generic.go:356] "Generic (PLEG): container finished" podID="f3217f73-aa32-4cf7-b9ea-771cab429a4f" containerID="1b2944d335822a0d2d66e7a1ca85834b5ac0fef33e16b0cc499a0d9298fc5471" exitCode=0 Jul 17 18:33:34.801284 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.801286 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:34.801521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:34.801320 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" event={"ID":"f3217f73-aa32-4cf7-b9ea-771cab429a4f","Type":"ContainerDied","Data":"1b2944d335822a0d2d66e7a1ca85834b5ac0fef33e16b0cc499a0d9298fc5471"} Jul 17 18:33:34.801569 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:34.801533 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:36.617205 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:36.616955 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:36.617674 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:36.616974 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:36.617674 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:36.617234 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kccpg" podUID="0f74a126-e2b2-4930-91f5-d8fab73ba4af" Jul 17 18:33:36.617674 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:36.616974 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:36.617674 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:36.617328 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z7rdv" podUID="30bd0145-a321-42a2-8607-83268d918215" Jul 17 18:33:36.617674 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:36.617386 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:33:36.808676 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:36.808626 2563 generic.go:356] "Generic (PLEG): container finished" podID="f3217f73-aa32-4cf7-b9ea-771cab429a4f" containerID="6d21e6131bc5db14d0f9871396c631e19682e7e6a2dc2b3d1cbe8ee0ebfc1586" exitCode=0 Jul 17 18:33:36.808909 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:36.808689 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" event={"ID":"f3217f73-aa32-4cf7-b9ea-771cab429a4f","Type":"ContainerDied","Data":"6d21e6131bc5db14d0f9871396c631e19682e7e6a2dc2b3d1cbe8ee0ebfc1586"} Jul 17 18:33:37.238331 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:37.238290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:37.238331 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:37.238333 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:37.238653 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:37.238438 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:37.238653 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:37.238482 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:37.238653 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:37.238531 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs podName:7e6936f2-9870-44cc-abd2-99ae76f575ff nodeName:}" failed. No retries permitted until 2026-07-17 18:34:09.238516707 +0000 UTC m=+66.142090561 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs") pod "network-metrics-daemon-6jqfb" (UID: "7e6936f2-9870-44cc-abd2-99ae76f575ff") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 18:33:37.238653 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:37.238545 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret podName:0f74a126-e2b2-4930-91f5-d8fab73ba4af nodeName:}" failed. No retries permitted until 2026-07-17 18:34:09.23853926 +0000 UTC m=+66.142113114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret") pod "global-pull-secret-syncer-kccpg" (UID: "0f74a126-e2b2-4930-91f5-d8fab73ba4af") : object "kube-system"/"original-pull-secret" not registered Jul 17 18:33:37.440336 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:37.440300 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmqf\" (UniqueName: \"kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf\") pod \"network-check-target-z7rdv\" (UID: \"30bd0145-a321-42a2-8607-83268d918215\") " pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:37.440526 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:37.440483 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 18:33:37.440526 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:37.440505 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 18:33:37.440526 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:37.440516 2563 projected.go:196] Error preparing data for projected volume kube-api-access-7cmqf for pod openshift-network-diagnostics/network-check-target-z7rdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:37.440628 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:37.440571 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf podName:30bd0145-a321-42a2-8607-83268d918215 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:09.440555178 +0000 UTC m=+66.344129032 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cmqf" (UniqueName: "kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf") pod "network-check-target-z7rdv" (UID: "30bd0145-a321-42a2-8607-83268d918215") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 18:33:38.474045 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.473969 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-220.ec2.internal" event="NodeReady" Jul 17 18:33:38.474406 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.474123 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 18:33:38.578811 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.578771 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67cc6846cb-j9mzl"] Jul 17 18:33:38.594710 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.594665 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.600125 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.600093 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 17 18:33:38.610040 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.610006 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 17 18:33:38.610232 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.610173 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 17 18:33:38.616283 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.616245 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:33:38.616475 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.616290 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:33:38.616475 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.616245 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:33:38.616726 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.616707 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 17 18:33:38.617506 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.617284 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h6x9z\"" Jul 17 18:33:38.639077 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.639050 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 18:33:38.644592 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.644562 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 18:33:38.644736 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.644592 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 18:33:38.648051 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.648030 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-st5cn\"" Jul 17 18:33:38.648882 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.648850 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 18:33:38.649344 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.649329 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67cc6846cb-j9mzl"] Jul 17 18:33:38.649826 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.649814 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kxtqg"] Jul 17 18:33:38.669755 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.669713 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gbnj9\"" Jul 17 18:33:38.671143 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.671120 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x2dl7"] Jul 17 18:33:38.671263 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.671235 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kxtqg" Jul 17 18:33:38.679146 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.679115 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 18:33:38.679502 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.679474 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lzjfd\"" Jul 17 18:33:38.679615 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.679578 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 18:33:38.684479 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.684423 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 18:33:38.699285 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.699252 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kxtqg"] Jul 17 18:33:38.699285 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.699289 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x2dl7"] Jul 17 18:33:38.699500 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.699410 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.702962 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.702938 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 18:33:38.703286 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.703264 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 18:33:38.703434 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.703415 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz2rf\"" Jul 17 18:33:38.750892 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.750801 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrf5z\" (UniqueName: \"kubernetes.io/projected/9ec210e0-c89d-4835-b168-f3708c29d87f-kube-api-access-rrf5z\") pod \"ingress-canary-kxtqg\" (UID: \"9ec210e0-c89d-4835-b168-f3708c29d87f\") " pod="openshift-ingress-canary/ingress-canary-kxtqg" Jul 17 18:33:38.750892 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.750863 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-certificates\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.751112 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.750921 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-image-registry-private-configuration\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.751112 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.750950 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-ca-trust-extracted\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.751112 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.750974 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xgvh\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-kube-api-access-5xgvh\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.751263 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.751129 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-bound-sa-token\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.751263 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.751199 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ec210e0-c89d-4835-b168-f3708c29d87f-cert\") pod \"ingress-canary-kxtqg\" (UID: \"9ec210e0-c89d-4835-b168-f3708c29d87f\") " pod="openshift-ingress-canary/ingress-canary-kxtqg" Jul 17 18:33:38.751263 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.751230 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.751397 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.751262 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-trusted-ca\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.751397 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.751289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-installation-pull-secrets\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.851834 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.851796 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-bound-sa-token\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.852042 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.851851 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ec210e0-c89d-4835-b168-f3708c29d87f-cert\") pod \"ingress-canary-kxtqg\" (UID: \"9ec210e0-c89d-4835-b168-f3708c29d87f\") " pod="openshift-ingress-canary/ingress-canary-kxtqg" Jul 17 18:33:38.852042 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.851881 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.852042 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.851909 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-trusted-ca\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.852042 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.851930 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-installation-pull-secrets\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.852042 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.851951 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.852042 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.851981 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrf5z\" (UniqueName: \"kubernetes.io/projected/9ec210e0-c89d-4835-b168-f3708c29d87f-kube-api-access-rrf5z\") pod \"ingress-canary-kxtqg\" (UID: \"9ec210e0-c89d-4835-b168-f3708c29d87f\") " pod="openshift-ingress-canary/ingress-canary-kxtqg" Jul 17 18:33:38.852042 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.852003 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tkc\" (UniqueName: \"kubernetes.io/projected/8a782ebc-a204-4a31-a122-bd141b81b66d-kube-api-access-t9tkc\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.852042 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.852026 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-certificates\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.852542 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.852061 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a782ebc-a204-4a31-a122-bd141b81b66d-tmp-dir\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.852542 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.852109 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-image-registry-private-configuration\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.852542 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.852168 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-ca-trust-extracted\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.852542 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.852189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xgvh\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-kube-api-access-5xgvh\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.852542 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.852225 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a782ebc-a204-4a31-a122-bd141b81b66d-config-volume\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.852542 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:38.852288 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:33:38.852542 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:38.852309 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc6846cb-j9mzl: secret "image-registry-tls" not found Jul 17 18:33:38.852542 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:38.852377 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls podName:ee7b110a-cf7a-4a6a-91f0-d49900cf3769 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:39.352358981 +0000 UTC m=+36.255932835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls") pod "image-registry-67cc6846cb-j9mzl" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769") : secret "image-registry-tls" not found Jul 17 18:33:38.852816 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.852801 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-certificates\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.853032 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.852994 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-trusted-ca\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.853344 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.853317 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-ca-trust-extracted\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.856897 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.856870 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-installation-pull-secrets\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.857066 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.856907 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ec210e0-c89d-4835-b168-f3708c29d87f-cert\") pod \"ingress-canary-kxtqg\" (UID: \"9ec210e0-c89d-4835-b168-f3708c29d87f\") " pod="openshift-ingress-canary/ingress-canary-kxtqg" Jul 17 18:33:38.860644 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.860616 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-image-registry-private-configuration\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.874986 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.874951 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-bound-sa-token\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.875125 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.875075 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xgvh\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-kube-api-access-5xgvh\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:38.875180 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.875127 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrf5z\" (UniqueName: \"kubernetes.io/projected/9ec210e0-c89d-4835-b168-f3708c29d87f-kube-api-access-rrf5z\") pod \"ingress-canary-kxtqg\" (UID: \"9ec210e0-c89d-4835-b168-f3708c29d87f\") " pod="openshift-ingress-canary/ingress-canary-kxtqg" Jul 17 18:33:38.953136 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.953102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.953346 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.953145 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tkc\" (UniqueName: \"kubernetes.io/projected/8a782ebc-a204-4a31-a122-bd141b81b66d-kube-api-access-t9tkc\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.953346 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.953173 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a782ebc-a204-4a31-a122-bd141b81b66d-tmp-dir\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.953346 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.953234 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a782ebc-a204-4a31-a122-bd141b81b66d-config-volume\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.953346 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:38.953284 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:33:38.953539 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:38.953354 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls podName:8a782ebc-a204-4a31-a122-bd141b81b66d nodeName:}" failed. No retries permitted until 2026-07-17 18:33:39.453333433 +0000 UTC m=+36.356907288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls") pod "dns-default-x2dl7" (UID: "8a782ebc-a204-4a31-a122-bd141b81b66d") : secret "dns-default-metrics-tls" not found Jul 17 18:33:38.953617 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.953600 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a782ebc-a204-4a31-a122-bd141b81b66d-tmp-dir\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.953843 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.953826 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a782ebc-a204-4a31-a122-bd141b81b66d-config-volume\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.965240 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.965208 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tkc\" (UniqueName: \"kubernetes.io/projected/8a782ebc-a204-4a31-a122-bd141b81b66d-kube-api-access-t9tkc\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:38.980047 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:38.980008 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kxtqg" Jul 17 18:33:39.204320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:39.204285 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kxtqg"] Jul 17 18:33:39.209999 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:39.209959 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec210e0_c89d_4835_b168_f3708c29d87f.slice/crio-f63879d3d601d36ed8b9fd23f70c24ba67aa9114658e4785734652bb5a7f2e06 WatchSource:0}: Error finding container f63879d3d601d36ed8b9fd23f70c24ba67aa9114658e4785734652bb5a7f2e06: Status 404 returned error can't find the container with id f63879d3d601d36ed8b9fd23f70c24ba67aa9114658e4785734652bb5a7f2e06 Jul 17 18:33:39.356643 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:39.356549 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:39.356822 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:39.356729 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:33:39.356822 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:39.356751 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc6846cb-j9mzl: secret "image-registry-tls" not found Jul 17 18:33:39.356822 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:39.356814 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls podName:ee7b110a-cf7a-4a6a-91f0-d49900cf3769 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:40.356795734 +0000 UTC m=+37.260369606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls") pod "image-registry-67cc6846cb-j9mzl" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769") : secret "image-registry-tls" not found Jul 17 18:33:39.457726 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:39.457684 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:39.457912 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:39.457839 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:33:39.457912 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:39.457905 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls podName:8a782ebc-a204-4a31-a122-bd141b81b66d nodeName:}" failed. No retries permitted until 2026-07-17 18:33:40.457890418 +0000 UTC m=+37.361464272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls") pod "dns-default-x2dl7" (UID: "8a782ebc-a204-4a31-a122-bd141b81b66d") : secret "dns-default-metrics-tls" not found Jul 17 18:33:39.817973 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:39.817932 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kxtqg" event={"ID":"9ec210e0-c89d-4835-b168-f3708c29d87f","Type":"ContainerStarted","Data":"f63879d3d601d36ed8b9fd23f70c24ba67aa9114658e4785734652bb5a7f2e06"} Jul 17 18:33:40.367663 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:40.367621 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:40.367857 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:40.367796 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:33:40.367857 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:40.367839 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc6846cb-j9mzl: secret "image-registry-tls" not found Jul 17 18:33:40.367950 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:40.367898 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls podName:ee7b110a-cf7a-4a6a-91f0-d49900cf3769 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:42.367881404 +0000 UTC m=+39.271455258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls") pod "image-registry-67cc6846cb-j9mzl" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769") : secret "image-registry-tls" not found Jul 17 18:33:40.468947 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:40.468905 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:40.469156 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:40.469075 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:33:40.469240 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:40.469160 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls podName:8a782ebc-a204-4a31-a122-bd141b81b66d nodeName:}" failed. No retries permitted until 2026-07-17 18:33:42.469133918 +0000 UTC m=+39.372707774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls") pod "dns-default-x2dl7" (UID: "8a782ebc-a204-4a31-a122-bd141b81b66d") : secret "dns-default-metrics-tls" not found Jul 17 18:33:42.386784 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:42.386509 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:42.387296 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:42.386651 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:33:42.387296 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:42.386831 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc6846cb-j9mzl: secret "image-registry-tls" not found Jul 17 18:33:42.387296 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:42.386891 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls podName:ee7b110a-cf7a-4a6a-91f0-d49900cf3769 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:46.386869175 +0000 UTC m=+43.290443029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls") pod "image-registry-67cc6846cb-j9mzl" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769") : secret "image-registry-tls" not found Jul 17 18:33:42.487903 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:42.487865 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:42.488099 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:42.488028 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:33:42.488161 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:42.488105 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls podName:8a782ebc-a204-4a31-a122-bd141b81b66d nodeName:}" failed. No retries permitted until 2026-07-17 18:33:46.488084689 +0000 UTC m=+43.391658547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls") pod "dns-default-x2dl7" (UID: "8a782ebc-a204-4a31-a122-bd141b81b66d") : secret "dns-default-metrics-tls" not found Jul 17 18:33:43.835138 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:43.835092 2563 generic.go:356] "Generic (PLEG): container finished" podID="f3217f73-aa32-4cf7-b9ea-771cab429a4f" containerID="fff7258334868426bf32d3846185c3ece78acf01f0d085b6ceed218a895ad142" exitCode=0 Jul 17 18:33:43.835575 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:43.835183 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" event={"ID":"f3217f73-aa32-4cf7-b9ea-771cab429a4f","Type":"ContainerDied","Data":"fff7258334868426bf32d3846185c3ece78acf01f0d085b6ceed218a895ad142"} Jul 17 18:33:43.836848 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:43.836819 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kxtqg" event={"ID":"9ec210e0-c89d-4835-b168-f3708c29d87f","Type":"ContainerStarted","Data":"6afeb60a961bf69e67e628775d08b43aa3257d445280813d41792f086e7ed56e"} Jul 17 18:33:43.896001 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:43.895947 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kxtqg" podStartSLOduration=2.121128941 podStartE2EDuration="5.895931138s" totalImagesPullingTime="3.774802197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:43.894741275 +0000 UTC m=+40.798315151" watchObservedRunningTime="2026-07-17 18:33:43.895931138 +0000 UTC m=+40.799505014" Jul 17 18:33:44.842681 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:44.842644 2563 generic.go:356] "Generic (PLEG): container finished" podID="f3217f73-aa32-4cf7-b9ea-771cab429a4f" containerID="264eb4a8f1fb0183752bd31317db992164b5e70565ebbe2e1d18dd3272ed039d" exitCode=0 Jul 17 18:33:44.843090 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:44.842734 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" event={"ID":"f3217f73-aa32-4cf7-b9ea-771cab429a4f","Type":"ContainerDied","Data":"264eb4a8f1fb0183752bd31317db992164b5e70565ebbe2e1d18dd3272ed039d"} Jul 17 18:33:45.848723 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:45.848676 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" event={"ID":"f3217f73-aa32-4cf7-b9ea-771cab429a4f","Type":"ContainerStarted","Data":"0427f13ca6d2c17af37d529f0c5dd9e8b05c8649234e143b2d46e862afd8db7c"} Jul 17 18:33:46.426542 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:46.426490 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:46.426709 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:46.426661 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:33:46.426709 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:46.426680 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc6846cb-j9mzl: secret "image-registry-tls" not found Jul 17 18:33:46.426791 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:46.426741 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls podName:ee7b110a-cf7a-4a6a-91f0-d49900cf3769 nodeName:}" failed. No retries permitted until 2026-07-17 18:33:54.426723861 +0000 UTC m=+51.330297717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls") pod "image-registry-67cc6846cb-j9mzl" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769") : secret "image-registry-tls" not found Jul 17 18:33:46.527548 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:46.527504 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:46.527704 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:46.527665 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:33:46.527768 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:46.527738 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls podName:8a782ebc-a204-4a31-a122-bd141b81b66d nodeName:}" failed. No retries permitted until 2026-07-17 18:33:54.527721022 +0000 UTC m=+51.431294876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls") pod "dns-default-x2dl7" (UID: "8a782ebc-a204-4a31-a122-bd141b81b66d") : secret "dns-default-metrics-tls" not found Jul 17 18:33:54.490432 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:54.490392 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:33:54.490902 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:54.490554 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:33:54.490902 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:54.490575 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc6846cb-j9mzl: secret "image-registry-tls" not found Jul 17 18:33:54.490902 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:54.490633 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls podName:ee7b110a-cf7a-4a6a-91f0-d49900cf3769 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:10.490617145 +0000 UTC m=+67.394191016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls") pod "image-registry-67cc6846cb-j9mzl" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769") : secret "image-registry-tls" not found Jul 17 18:33:54.591321 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:54.591276 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:33:54.591563 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:54.591469 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:33:54.591563 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:33:54.591548 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls podName:8a782ebc-a204-4a31-a122-bd141b81b66d nodeName:}" failed. No retries permitted until 2026-07-17 18:34:10.591528852 +0000 UTC m=+67.495102727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls") pod "dns-default-x2dl7" (UID: "8a782ebc-a204-4a31-a122-bd141b81b66d") : secret "dns-default-metrics-tls" not found Jul 17 18:33:58.808438 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.808378 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5kmz6" podStartSLOduration=20.779238307 podStartE2EDuration="54.808353439s" totalImagesPullingTime="34.029115132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:33:45.882170676 +0000 UTC m=+42.785744552" watchObservedRunningTime="2026-07-17 18:33:58.808353439 +0000 UTC m=+55.711927316" Jul 17 18:33:58.808909 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.808754 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24"] Jul 17 18:33:58.852958 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.852922 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24"] Jul 17 18:33:58.853149 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.853055 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:58.861329 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.861309 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 17 18:33:58.862132 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.862117 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 17 18:33:58.862205 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.862176 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 17 18:33:58.864300 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.864281 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 17 18:33:58.923632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.923588 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6t4l\" (UniqueName: \"kubernetes.io/projected/e706fba2-dcfe-4886-aa60-2edb19cd22ee-kube-api-access-x6t4l\") pod \"klusterlet-addon-workmgr-596ddd7bc4-f7n24\" (UID: \"e706fba2-dcfe-4886-aa60-2edb19cd22ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:58.923632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.923638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e706fba2-dcfe-4886-aa60-2edb19cd22ee-tmp\") pod \"klusterlet-addon-workmgr-596ddd7bc4-f7n24\" (UID: \"e706fba2-dcfe-4886-aa60-2edb19cd22ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:58.923881 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:58.923659 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e706fba2-dcfe-4886-aa60-2edb19cd22ee-klusterlet-config\") pod \"klusterlet-addon-workmgr-596ddd7bc4-f7n24\" (UID: \"e706fba2-dcfe-4886-aa60-2edb19cd22ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:59.024885 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:59.024844 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6t4l\" (UniqueName: \"kubernetes.io/projected/e706fba2-dcfe-4886-aa60-2edb19cd22ee-kube-api-access-x6t4l\") pod \"klusterlet-addon-workmgr-596ddd7bc4-f7n24\" (UID: \"e706fba2-dcfe-4886-aa60-2edb19cd22ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:59.025092 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:59.024906 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e706fba2-dcfe-4886-aa60-2edb19cd22ee-tmp\") pod \"klusterlet-addon-workmgr-596ddd7bc4-f7n24\" (UID: \"e706fba2-dcfe-4886-aa60-2edb19cd22ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:59.025092 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:59.024936 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e706fba2-dcfe-4886-aa60-2edb19cd22ee-klusterlet-config\") pod \"klusterlet-addon-workmgr-596ddd7bc4-f7n24\" (UID: \"e706fba2-dcfe-4886-aa60-2edb19cd22ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:59.025268 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:59.025251 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e706fba2-dcfe-4886-aa60-2edb19cd22ee-tmp\") pod \"klusterlet-addon-workmgr-596ddd7bc4-f7n24\" (UID: \"e706fba2-dcfe-4886-aa60-2edb19cd22ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:59.028695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:59.028659 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e706fba2-dcfe-4886-aa60-2edb19cd22ee-klusterlet-config\") pod \"klusterlet-addon-workmgr-596ddd7bc4-f7n24\" (UID: \"e706fba2-dcfe-4886-aa60-2edb19cd22ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:59.035048 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:59.035019 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6t4l\" (UniqueName: \"kubernetes.io/projected/e706fba2-dcfe-4886-aa60-2edb19cd22ee-kube-api-access-x6t4l\") pod \"klusterlet-addon-workmgr-596ddd7bc4-f7n24\" (UID: \"e706fba2-dcfe-4886-aa60-2edb19cd22ee\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:59.171706 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:59.171668 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:33:59.335326 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:59.335291 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24"] Jul 17 18:33:59.339060 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:33:59.339031 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode706fba2_dcfe_4886_aa60_2edb19cd22ee.slice/crio-64799055faade174be33d61485a83c3924176371727a2e2cee756a530c11e45f WatchSource:0}: Error finding container 64799055faade174be33d61485a83c3924176371727a2e2cee756a530c11e45f: Status 404 returned error can't find the container with id 64799055faade174be33d61485a83c3924176371727a2e2cee756a530c11e45f Jul 17 18:33:59.885330 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:33:59.885288 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" event={"ID":"e706fba2-dcfe-4886-aa60-2edb19cd22ee","Type":"ContainerStarted","Data":"64799055faade174be33d61485a83c3924176371727a2e2cee756a530c11e45f"} Jul 17 18:34:03.897595 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:03.897557 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" event={"ID":"e706fba2-dcfe-4886-aa60-2edb19cd22ee","Type":"ContainerStarted","Data":"7dd3700685cb881f67caa18b1e14164a259ed6c590504574fbafd98cd406085a"} Jul 17 18:34:03.897983 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:03.897776 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:34:03.899488 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:03.899464 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:34:03.928137 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:03.928090 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" podStartSLOduration=2.00155531 podStartE2EDuration="5.928075176s" totalImagesPullingTime="3.926519866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:03.927751208 +0000 UTC m=+60.831325085" watchObservedRunningTime="2026-07-17 18:34:03.928075176 +0000 UTC m=+60.831649052" Jul 17 18:34:04.818038 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:04.818013 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8jqr2" Jul 17 18:34:09.305056 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.305002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:34:09.305056 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.305061 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:34:09.308334 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.308287 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 18:34:09.308608 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.308591 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 18:34:09.316158 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:09.316131 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 18:34:09.316317 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:09.316215 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs podName:7e6936f2-9870-44cc-abd2-99ae76f575ff nodeName:}" failed. No retries permitted until 2026-07-17 18:35:13.316192271 +0000 UTC m=+130.219766139 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs") pod "network-metrics-daemon-6jqfb" (UID: "7e6936f2-9870-44cc-abd2-99ae76f575ff") : secret "metrics-daemon-secret" not found Jul 17 18:34:09.318567 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.318541 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f74a126-e2b2-4930-91f5-d8fab73ba4af-original-pull-secret\") pod \"global-pull-secret-syncer-kccpg\" (UID: \"0f74a126-e2b2-4930-91f5-d8fab73ba4af\") " pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:34:09.507044 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.507001 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmqf\" (UniqueName: \"kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf\") pod \"network-check-target-z7rdv\" (UID: \"30bd0145-a321-42a2-8607-83268d918215\") " pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:34:09.509576 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.509557 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 18:34:09.520218 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.520194 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 18:34:09.530813 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.530786 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmqf\" (UniqueName: \"kubernetes.io/projected/30bd0145-a321-42a2-8607-83268d918215-kube-api-access-7cmqf\") pod \"network-check-target-z7rdv\" (UID: \"30bd0145-a321-42a2-8607-83268d918215\") " pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:34:09.540296 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.540264 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-st5cn\"" Jul 17 18:34:09.542301 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.542281 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kccpg" Jul 17 18:34:09.546629 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.546611 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:34:09.689987 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.689943 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kccpg"] Jul 17 18:34:09.694096 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:34:09.694067 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f74a126_e2b2_4930_91f5_d8fab73ba4af.slice/crio-0aa0f7a87b264effe4a9b35b677396584edaae690b22bf0076dd3b7484492ab0 WatchSource:0}: Error finding container 0aa0f7a87b264effe4a9b35b677396584edaae690b22bf0076dd3b7484492ab0: Status 404 returned error can't find the container with id 0aa0f7a87b264effe4a9b35b677396584edaae690b22bf0076dd3b7484492ab0 Jul 17 18:34:09.711331 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.711305 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z7rdv"] Jul 17 18:34:09.913376 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.913338 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7rdv" event={"ID":"30bd0145-a321-42a2-8607-83268d918215","Type":"ContainerStarted","Data":"5d9afe3e3291ed726aa20c3c2b7b2c0235686817733c6c77f3f7a729a6db26c5"} Jul 17 18:34:09.914510 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:09.914485 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kccpg" event={"ID":"0f74a126-e2b2-4930-91f5-d8fab73ba4af","Type":"ContainerStarted","Data":"0aa0f7a87b264effe4a9b35b677396584edaae690b22bf0076dd3b7484492ab0"} Jul 17 18:34:10.514848 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:10.514808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:34:10.515432 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:10.514936 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:34:10.515432 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:10.514956 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc6846cb-j9mzl: secret "image-registry-tls" not found Jul 17 18:34:10.515432 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:10.515018 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls podName:ee7b110a-cf7a-4a6a-91f0-d49900cf3769 nodeName:}" failed. No retries permitted until 2026-07-17 18:34:42.515003141 +0000 UTC m=+99.418576995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls") pod "image-registry-67cc6846cb-j9mzl" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769") : secret "image-registry-tls" not found Jul 17 18:34:10.615593 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:10.615554 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:34:10.615800 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:10.615710 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:34:10.615800 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:10.615779 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls podName:8a782ebc-a204-4a31-a122-bd141b81b66d nodeName:}" failed. No retries permitted until 2026-07-17 18:34:42.615760998 +0000 UTC m=+99.519334865 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls") pod "dns-default-x2dl7" (UID: "8a782ebc-a204-4a31-a122-bd141b81b66d") : secret "dns-default-metrics-tls" not found Jul 17 18:34:13.928724 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:13.928678 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z7rdv" event={"ID":"30bd0145-a321-42a2-8607-83268d918215","Type":"ContainerStarted","Data":"72c3d7067d21f497dbaa2dd47f43cf4704c95aca7435b48cd5cf25c9cedc7b33"} Jul 17 18:34:13.929156 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:13.928787 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:34:13.957212 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:13.957137 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z7rdv" podStartSLOduration=66.479984769 podStartE2EDuration="1m9.957113653s" totalImagesPullingTime="3.477128884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:13.954861633 +0000 UTC m=+70.858435510" watchObservedRunningTime="2026-07-17 18:34:13.957113653 +0000 UTC m=+70.860687530" Jul 17 18:34:15.935839 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:15.935797 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kccpg" event={"ID":"0f74a126-e2b2-4930-91f5-d8fab73ba4af","Type":"ContainerStarted","Data":"95d53d9877cd3eadff9a1c586bb6cb009818551910a2b5319957fa4578a87919"} Jul 17 18:34:15.952927 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:15.952875 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kccpg" podStartSLOduration=66.56740968 podStartE2EDuration="1m11.952859011s" totalImagesPullingTime="5.385449331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:34:15.952341102 +0000 UTC m=+72.855914979" watchObservedRunningTime="2026-07-17 18:34:15.952859011 +0000 UTC m=+72.856432894" Jul 17 18:34:42.548089 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:42.547948 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:34:42.548575 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:42.548112 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:34:42.548575 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:42.548131 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc6846cb-j9mzl: secret "image-registry-tls" not found Jul 17 18:34:42.548575 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:42.548211 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls podName:ee7b110a-cf7a-4a6a-91f0-d49900cf3769 nodeName:}" failed. No retries permitted until 2026-07-17 18:35:46.548196042 +0000 UTC m=+163.451769895 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls") pod "image-registry-67cc6846cb-j9mzl" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769") : secret "image-registry-tls" not found Jul 17 18:34:42.649184 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:42.649147 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:34:42.649359 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:42.649296 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:34:42.649359 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:34:42.649355 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls podName:8a782ebc-a204-4a31-a122-bd141b81b66d nodeName:}" failed. No retries permitted until 2026-07-17 18:35:46.649338147 +0000 UTC m=+163.552912022 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls") pod "dns-default-x2dl7" (UID: "8a782ebc-a204-4a31-a122-bd141b81b66d") : secret "dns-default-metrics-tls" not found Jul 17 18:34:44.934156 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:34:44.934123 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z7rdv" Jul 17 18:35:13.380469 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:35:13.380389 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:35:13.381062 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:35:13.380571 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 18:35:13.381062 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:35:13.380662 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs podName:7e6936f2-9870-44cc-abd2-99ae76f575ff nodeName:}" failed. No retries permitted until 2026-07-17 18:37:15.380637448 +0000 UTC m=+252.284211305 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs") pod "network-metrics-daemon-6jqfb" (UID: "7e6936f2-9870-44cc-abd2-99ae76f575ff") : secret "metrics-daemon-secret" not found Jul 17 18:35:41.606966 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:35:41.606915 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" podUID="ee7b110a-cf7a-4a6a-91f0-d49900cf3769" Jul 17 18:35:41.629286 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:35:41.629239 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:35:41.711788 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:35:41.711733 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-x2dl7" podUID="8a782ebc-a204-4a31-a122-bd141b81b66d" Jul 17 18:35:42.160099 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:35:42.160068 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:35:46.606734 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:35:46.606692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:35:46.607170 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:35:46.606808 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 17 18:35:46.607170 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:35:46.606819 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67cc6846cb-j9mzl: secret "image-registry-tls" not found Jul 17 18:35:46.607170 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:35:46.606871 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls podName:ee7b110a-cf7a-4a6a-91f0-d49900cf3769 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:48.606855565 +0000 UTC m=+285.510429418 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls") pod "image-registry-67cc6846cb-j9mzl" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769") : secret "image-registry-tls" not found Jul 17 18:35:46.707257 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:35:46.707208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:35:46.707378 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:35:46.707363 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 17 18:35:46.707468 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:35:46.707436 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls podName:8a782ebc-a204-4a31-a122-bd141b81b66d nodeName:}" failed. No retries permitted until 2026-07-17 18:37:48.707416258 +0000 UTC m=+285.610990133 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls") pod "dns-default-x2dl7" (UID: "8a782ebc-a204-4a31-a122-bd141b81b66d") : secret "dns-default-metrics-tls" not found Jul 17 18:35:54.616338 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:35:54.616233 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:35:56.616308 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:35:56.616249 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x2dl7" Jul 17 18:36:03.898333 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:36:03.898276 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" podUID="e706fba2-dcfe-4886-aa60-2edb19cd22ee" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused" Jul 17 18:36:04.213674 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:36:04.213637 2563 generic.go:356] "Generic (PLEG): container finished" podID="e706fba2-dcfe-4886-aa60-2edb19cd22ee" containerID="7dd3700685cb881f67caa18b1e14164a259ed6c590504574fbafd98cd406085a" exitCode=1 Jul 17 18:36:04.213849 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:36:04.213713 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" event={"ID":"e706fba2-dcfe-4886-aa60-2edb19cd22ee","Type":"ContainerDied","Data":"7dd3700685cb881f67caa18b1e14164a259ed6c590504574fbafd98cd406085a"} Jul 17 18:36:04.214058 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:36:04.214045 2563 scope.go:117] "RemoveContainer" containerID="7dd3700685cb881f67caa18b1e14164a259ed6c590504574fbafd98cd406085a" Jul 17 18:36:05.218101 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:36:05.218065 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" event={"ID":"e706fba2-dcfe-4886-aa60-2edb19cd22ee","Type":"ContainerStarted","Data":"988866a50bc0bea64983d4e9fbe0e18786a4a5a7b1f402779d9de3bd77cfabe4"} Jul 17 18:36:05.218525 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:36:05.218397 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:36:05.218945 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:36:05.218929 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:37:15.453625 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:15.453562 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:37:15.454132 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:15.453709 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 17 18:37:15.454132 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:15.453782 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs podName:7e6936f2-9870-44cc-abd2-99ae76f575ff nodeName:}" failed. No retries permitted until 2026-07-17 18:39:17.453766017 +0000 UTC m=+374.357339870 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs") pod "network-metrics-daemon-6jqfb" (UID: "7e6936f2-9870-44cc-abd2-99ae76f575ff") : secret "metrics-daemon-secret" not found Jul 17 18:37:16.662876 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.662833 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t"] Jul 17 18:37:16.664723 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.664705 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t" Jul 17 18:37:16.667313 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.667286 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 17 18:37:16.667486 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.667287 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:37:16.668129 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.668107 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fntr5\"" Jul 17 18:37:16.672141 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.672122 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j"] Jul 17 18:37:16.673938 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.673917 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j" Jul 17 18:37:16.676107 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:16.676080 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"network-diagnostics-dockercfg-w275w\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-network-diagnostics\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-w275w\"" type="*v1.Secret" Jul 17 18:37:16.691979 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.691933 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t"] Jul 17 18:37:16.692669 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.692648 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j"] Jul 17 18:37:16.761746 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.761704 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lcvf\" (UniqueName: \"kubernetes.io/projected/9d96a286-224f-4ff5-a5d1-79afd0d51a22-kube-api-access-6lcvf\") pod \"volume-data-source-validator-78cf4c6f6-d2m7t\" (UID: \"9d96a286-224f-4ff5-a5d1-79afd0d51a22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t" Jul 17 18:37:16.761937 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.761762 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bmxk\" (UniqueName: \"kubernetes.io/projected/28f227a4-da33-4f72-904a-008edb2f68f6-kube-api-access-5bmxk\") pod \"network-check-source-7bc5b6bf85-rk78j\" (UID: \"28f227a4-da33-4f72-904a-008edb2f68f6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j" Jul 17 18:37:16.770027 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.769993 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-57c488db8b-cxf75"] Jul 17 18:37:16.771914 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.771889 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w"] Jul 17 18:37:16.772077 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.772013 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.773701 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.773676 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:16.777932 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.777909 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:37:16.779202 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.779181 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pgngq\"" Jul 17 18:37:16.779479 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.779441 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 18:37:16.779538 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.779469 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 17 18:37:16.782022 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.782000 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 17 18:37:16.783989 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.782521 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 17 18:37:16.783989 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.782687 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 17 18:37:16.783989 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.782700 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 17 18:37:16.783989 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.782816 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 17 18:37:16.783989 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.783350 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-v7zrp\"" Jul 17 18:37:16.783989 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.783399 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 17 18:37:16.790353 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.790173 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 18:37:16.790522 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.790374 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w"] Jul 17 18:37:16.791308 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.791287 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57c488db8b-cxf75"] Jul 17 18:37:16.851589 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.851554 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n"] Jul 17 18:37:16.853578 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.853553 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:16.856501 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.856479 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-k4jf5\"" Jul 17 18:37:16.856501 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.856492 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 17 18:37:16.856670 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.856493 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 17 18:37:16.862174 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862148 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-stats-auth\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.862288 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862184 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:16.862288 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862216 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-default-certificate\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.862359 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ff9293e-8e04-4639-b2e8-4e806d121427-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-mt27w\" (UID: \"3ff9293e-8e04-4639-b2e8-4e806d121427\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:16.862359 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862339 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lcvf\" (UniqueName: \"kubernetes.io/projected/9d96a286-224f-4ff5-a5d1-79afd0d51a22-kube-api-access-6lcvf\") pod \"volume-data-source-validator-78cf4c6f6-d2m7t\" (UID: \"9d96a286-224f-4ff5-a5d1-79afd0d51a22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t" Jul 17 18:37:16.862423 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862381 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d1af3565-93a3-4ce2-933d-c527fc2102ca-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:16.862423 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bmxk\" (UniqueName: \"kubernetes.io/projected/28f227a4-da33-4f72-904a-008edb2f68f6-kube-api-access-5bmxk\") pod \"network-check-source-7bc5b6bf85-rk78j\" (UID: \"28f227a4-da33-4f72-904a-008edb2f68f6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j" Jul 17 18:37:16.862423 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862417 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ff9293e-8e04-4639-b2e8-4e806d121427-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-mt27w\" (UID: \"3ff9293e-8e04-4639-b2e8-4e806d121427\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:16.862567 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862438 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.862567 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862476 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsg6w\" (UniqueName: \"kubernetes.io/projected/1c73c210-fc62-4860-93e4-c352a8021182-kube-api-access-fsg6w\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.862567 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862525 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.862687 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.862575 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkrf\" (UniqueName: \"kubernetes.io/projected/3ff9293e-8e04-4639-b2e8-4e806d121427-kube-api-access-jgkrf\") pod \"kube-storage-version-migrator-operator-7658d9b749-mt27w\" (UID: \"3ff9293e-8e04-4639-b2e8-4e806d121427\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:16.871089 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.871056 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n"] Jul 17 18:37:16.878410 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.878379 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lcvf\" (UniqueName: \"kubernetes.io/projected/9d96a286-224f-4ff5-a5d1-79afd0d51a22-kube-api-access-6lcvf\") pod \"volume-data-source-validator-78cf4c6f6-d2m7t\" (UID: \"9d96a286-224f-4ff5-a5d1-79afd0d51a22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t" Jul 17 18:37:16.878572 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.878505 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bmxk\" (UniqueName: \"kubernetes.io/projected/28f227a4-da33-4f72-904a-008edb2f68f6-kube-api-access-5bmxk\") pod \"network-check-source-7bc5b6bf85-rk78j\" (UID: \"28f227a4-da33-4f72-904a-008edb2f68f6\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j" Jul 17 18:37:16.963044 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.962957 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.963044 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.963009 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkrf\" (UniqueName: \"kubernetes.io/projected/3ff9293e-8e04-4639-b2e8-4e806d121427-kube-api-access-jgkrf\") pod \"kube-storage-version-migrator-operator-7658d9b749-mt27w\" (UID: \"3ff9293e-8e04-4639-b2e8-4e806d121427\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:16.963044 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.963041 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-stats-auth\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.963324 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.963058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:16.963324 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.963080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-default-certificate\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.963324 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:16.963092 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 18:37:16.963324 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:16.963180 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs podName:1c73c210-fc62-4860-93e4-c352a8021182 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:17.463164461 +0000 UTC m=+254.366738314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs") pod "router-default-57c488db8b-cxf75" (UID: "1c73c210-fc62-4860-93e4-c352a8021182") : secret "router-metrics-certs-default" not found Jul 17 18:37:16.963324 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:16.963211 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 17 18:37:16.963324 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.963259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ff9293e-8e04-4639-b2e8-4e806d121427-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-mt27w\" (UID: \"3ff9293e-8e04-4639-b2e8-4e806d121427\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:16.963324 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:16.963302 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert podName:d1af3565-93a3-4ce2-933d-c527fc2102ca nodeName:}" failed. No retries permitted until 2026-07-17 18:37:17.463282328 +0000 UTC m=+254.366856203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-ghl9n" (UID: "d1af3565-93a3-4ce2-933d-c527fc2102ca") : secret "networking-console-plugin-cert" not found Jul 17 18:37:16.963717 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.963345 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d1af3565-93a3-4ce2-933d-c527fc2102ca-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:16.963717 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.963395 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ff9293e-8e04-4639-b2e8-4e806d121427-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-mt27w\" (UID: \"3ff9293e-8e04-4639-b2e8-4e806d121427\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:16.963717 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.963431 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.963717 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.963483 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsg6w\" (UniqueName: \"kubernetes.io/projected/1c73c210-fc62-4860-93e4-c352a8021182-kube-api-access-fsg6w\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.963717 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:16.963622 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle podName:1c73c210-fc62-4860-93e4-c352a8021182 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:17.463584139 +0000 UTC m=+254.367158092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle") pod "router-default-57c488db8b-cxf75" (UID: "1c73c210-fc62-4860-93e4-c352a8021182") : configmap references non-existent config key: service-ca.crt Jul 17 18:37:16.964188 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.964167 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ff9293e-8e04-4639-b2e8-4e806d121427-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-mt27w\" (UID: \"3ff9293e-8e04-4639-b2e8-4e806d121427\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:16.964382 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.964361 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d1af3565-93a3-4ce2-933d-c527fc2102ca-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:16.965683 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.965656 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ff9293e-8e04-4639-b2e8-4e806d121427-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-mt27w\" (UID: \"3ff9293e-8e04-4639-b2e8-4e806d121427\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:16.965799 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.965780 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-stats-auth\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.965839 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.965806 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-default-certificate\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.972766 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.972732 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkrf\" (UniqueName: \"kubernetes.io/projected/3ff9293e-8e04-4639-b2e8-4e806d121427-kube-api-access-jgkrf\") pod \"kube-storage-version-migrator-operator-7658d9b749-mt27w\" (UID: \"3ff9293e-8e04-4639-b2e8-4e806d121427\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:16.972923 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.972881 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsg6w\" (UniqueName: \"kubernetes.io/projected/1c73c210-fc62-4860-93e4-c352a8021182-kube-api-access-fsg6w\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:16.974061 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:16.974043 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t" Jul 17 18:37:17.092386 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.092352 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" Jul 17 18:37:17.113871 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.113842 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t"] Jul 17 18:37:17.118370 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:37:17.118343 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d96a286_224f_4ff5_a5d1_79afd0d51a22.slice/crio-57e4942d1369d0078f8e409b26e3486866246a7091f3cda94eed67d40045ae8b WatchSource:0}: Error finding container 57e4942d1369d0078f8e409b26e3486866246a7091f3cda94eed67d40045ae8b: Status 404 returned error can't find the container with id 57e4942d1369d0078f8e409b26e3486866246a7091f3cda94eed67d40045ae8b Jul 17 18:37:17.228216 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.228139 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w"] Jul 17 18:37:17.231414 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:37:17.231388 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff9293e_8e04_4639_b2e8_4e806d121427.slice/crio-72b15927afe8ae7a614ed1f8f81011e987583bfd2362a0c4f9491294555789b9 WatchSource:0}: Error finding container 72b15927afe8ae7a614ed1f8f81011e987583bfd2362a0c4f9491294555789b9: Status 404 returned error can't find the container with id 72b15927afe8ae7a614ed1f8f81011e987583bfd2362a0c4f9491294555789b9 Jul 17 18:37:17.395243 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.395208 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" event={"ID":"3ff9293e-8e04-4639-b2e8-4e806d121427","Type":"ContainerStarted","Data":"72b15927afe8ae7a614ed1f8f81011e987583bfd2362a0c4f9491294555789b9"} Jul 17 18:37:17.396345 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.396322 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t" event={"ID":"9d96a286-224f-4ff5-a5d1-79afd0d51a22","Type":"ContainerStarted","Data":"57e4942d1369d0078f8e409b26e3486866246a7091f3cda94eed67d40045ae8b"} Jul 17 18:37:17.467248 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.467217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:17.467386 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.467260 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:17.467386 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.467295 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:17.467386 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:17.467380 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 17 18:37:17.467540 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:17.467382 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 18:37:17.467540 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:17.467407 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle podName:1c73c210-fc62-4860-93e4-c352a8021182 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:18.467385637 +0000 UTC m=+255.370959505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle") pod "router-default-57c488db8b-cxf75" (UID: "1c73c210-fc62-4860-93e4-c352a8021182") : configmap references non-existent config key: service-ca.crt Jul 17 18:37:17.467540 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:17.467433 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs podName:1c73c210-fc62-4860-93e4-c352a8021182 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:18.46742077 +0000 UTC m=+255.370994623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs") pod "router-default-57c488db8b-cxf75" (UID: "1c73c210-fc62-4860-93e4-c352a8021182") : secret "router-metrics-certs-default" not found Jul 17 18:37:17.467540 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:17.467471 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert podName:d1af3565-93a3-4ce2-933d-c527fc2102ca nodeName:}" failed. No retries permitted until 2026-07-17 18:37:18.467464353 +0000 UTC m=+255.371038207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-ghl9n" (UID: "d1af3565-93a3-4ce2-933d-c527fc2102ca") : secret "networking-console-plugin-cert" not found Jul 17 18:37:17.816827 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.816799 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-w275w\"" Jul 17 18:37:17.822984 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.822942 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j" Jul 17 18:37:17.979519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:17.979484 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j"] Jul 17 18:37:17.982337 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:37:17.982305 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f227a4_da33_4f72_904a_008edb2f68f6.slice/crio-51780151d21b3b1b134317be273e740294d21e0d4acc9585505394304b0f3008 WatchSource:0}: Error finding container 51780151d21b3b1b134317be273e740294d21e0d4acc9585505394304b0f3008: Status 404 returned error can't find the container with id 51780151d21b3b1b134317be273e740294d21e0d4acc9585505394304b0f3008 Jul 17 18:37:18.401712 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:18.401679 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j" event={"ID":"28f227a4-da33-4f72-904a-008edb2f68f6","Type":"ContainerStarted","Data":"29eb3a81c4bd4c4f07cf4be5604a279fc477a68d32d2bfe6ccd845d77c22fbb8"} Jul 17 18:37:18.401890 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:18.401726 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j" event={"ID":"28f227a4-da33-4f72-904a-008edb2f68f6","Type":"ContainerStarted","Data":"51780151d21b3b1b134317be273e740294d21e0d4acc9585505394304b0f3008"} Jul 17 18:37:18.420658 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:18.420531 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-rk78j" podStartSLOduration=2.420510449 podStartE2EDuration="2.420510449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:18.419179082 +0000 UTC m=+255.322752959" watchObservedRunningTime="2026-07-17 18:37:18.420510449 +0000 UTC m=+255.324084326" Jul 17 18:37:18.475897 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:18.475856 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:18.476081 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:18.476016 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle podName:1c73c210-fc62-4860-93e4-c352a8021182 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:20.475995438 +0000 UTC m=+257.379569293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle") pod "router-default-57c488db8b-cxf75" (UID: "1c73c210-fc62-4860-93e4-c352a8021182") : configmap references non-existent config key: service-ca.crt Jul 17 18:37:18.476245 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:18.476223 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:18.476342 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:18.476313 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:18.476390 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:18.476353 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 18:37:18.476457 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:18.476413 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs podName:1c73c210-fc62-4860-93e4-c352a8021182 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:20.476394493 +0000 UTC m=+257.379968349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs") pod "router-default-57c488db8b-cxf75" (UID: "1c73c210-fc62-4860-93e4-c352a8021182") : secret "router-metrics-certs-default" not found Jul 17 18:37:18.476457 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:18.476419 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 17 18:37:18.476598 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:18.476481 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert podName:d1af3565-93a3-4ce2-933d-c527fc2102ca nodeName:}" failed. No retries permitted until 2026-07-17 18:37:20.47646698 +0000 UTC m=+257.380040835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-ghl9n" (UID: "d1af3565-93a3-4ce2-933d-c527fc2102ca") : secret "networking-console-plugin-cert" not found Jul 17 18:37:19.406896 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:19.406847 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t" event={"ID":"9d96a286-224f-4ff5-a5d1-79afd0d51a22","Type":"ContainerStarted","Data":"5b48bfe2fe60d20564ee25eb75aa9bc7147dfa9c6e7ddf475683643d8badea86"} Jul 17 18:37:19.426746 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:19.426683 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-d2m7t" podStartSLOduration=2.034991425 podStartE2EDuration="3.426657854s" totalImagesPullingTime="1.391666429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:19.425303894 +0000 UTC m=+256.328877774" watchObservedRunningTime="2026-07-17 18:37:19.426657854 +0000 UTC m=+256.330231755" Jul 17 18:37:20.411957 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:20.411917 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" event={"ID":"3ff9293e-8e04-4639-b2e8-4e806d121427","Type":"ContainerStarted","Data":"496a3b63a3a4daf633981517749151828b3a2fe1d667566bb1fc42a04ffe04b2"} Jul 17 18:37:20.438381 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:20.438323 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" podStartSLOduration=1.9911965820000002 podStartE2EDuration="4.438302754s" totalImagesPullingTime="2.447106172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:20.438235578 +0000 UTC m=+257.341809454" watchObservedRunningTime="2026-07-17 18:37:20.438302754 +0000 UTC m=+257.341876631" Jul 17 18:37:20.491980 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:20.491943 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:20.492144 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:20.492064 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 17 18:37:20.492144 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:20.492140 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert podName:d1af3565-93a3-4ce2-933d-c527fc2102ca nodeName:}" failed. No retries permitted until 2026-07-17 18:37:24.492120165 +0000 UTC m=+261.395694022 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-ghl9n" (UID: "d1af3565-93a3-4ce2-933d-c527fc2102ca") : secret "networking-console-plugin-cert" not found Jul 17 18:37:20.492301 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:20.492274 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:20.492354 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:20.492339 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:20.492506 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:20.492440 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 18:37:20.492646 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:20.492516 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs podName:1c73c210-fc62-4860-93e4-c352a8021182 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:24.492501524 +0000 UTC m=+261.396075380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs") pod "router-default-57c488db8b-cxf75" (UID: "1c73c210-fc62-4860-93e4-c352a8021182") : secret "router-metrics-certs-default" not found Jul 17 18:37:20.492646 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:20.492575 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle podName:1c73c210-fc62-4860-93e4-c352a8021182 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:24.492553888 +0000 UTC m=+261.396127756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle") pod "router-default-57c488db8b-cxf75" (UID: "1c73c210-fc62-4860-93e4-c352a8021182") : configmap references non-existent config key: service-ca.crt Jul 17 18:37:21.386431 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.386397 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gn8pw"] Jul 17 18:37:21.389708 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.389685 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.392388 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.392361 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 18:37:21.392568 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.392471 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 18:37:21.392661 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.392649 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 18:37:21.392874 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.392857 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 18:37:21.393171 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.393154 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jvlrf\"" Jul 17 18:37:21.398274 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.398252 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.398387 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.398284 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9dbad995-ab40-421a-bb52-4a6723f31ffd-crio-socket\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.398387 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.398382 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.398520 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.398440 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nfjq\" (UniqueName: \"kubernetes.io/projected/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-api-access-9nfjq\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.398570 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.398526 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dbad995-ab40-421a-bb52-4a6723f31ffd-data-volume\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.409581 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.409553 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gn8pw"] Jul 17 18:37:21.499530 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.499483 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.500010 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.499555 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nfjq\" (UniqueName: \"kubernetes.io/projected/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-api-access-9nfjq\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.500010 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:21.499588 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:37:21.500010 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.499587 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dbad995-ab40-421a-bb52-4a6723f31ffd-data-volume\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.500010 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:21.499650 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm podName:9dbad995-ab40-421a-bb52-4a6723f31ffd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:21.999633539 +0000 UTC m=+258.903207397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gn8pw" (UID: "9dbad995-ab40-421a-bb52-4a6723f31ffd") : configmap "kube-rbac-proxy" not found Jul 17 18:37:21.500010 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.499682 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.500010 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.499707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9dbad995-ab40-421a-bb52-4a6723f31ffd-crio-socket\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.500010 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:21.499819 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:37:21.500010 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.499828 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9dbad995-ab40-421a-bb52-4a6723f31ffd-crio-socket\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.500010 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:21.499864 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls podName:9dbad995-ab40-421a-bb52-4a6723f31ffd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:21.999850477 +0000 UTC m=+258.903424345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gn8pw" (UID: "9dbad995-ab40-421a-bb52-4a6723f31ffd") : secret "insights-runtime-extractor-tls" not found Jul 17 18:37:21.500010 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.499899 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dbad995-ab40-421a-bb52-4a6723f31ffd-data-volume\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:21.514591 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:21.514560 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nfjq\" (UniqueName: \"kubernetes.io/projected/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-api-access-9nfjq\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:22.004079 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:22.004038 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:22.004270 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:22.004096 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:22.004270 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:22.004179 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:37:22.004270 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:22.004205 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:37:22.004270 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:22.004251 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm podName:9dbad995-ab40-421a-bb52-4a6723f31ffd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:23.004231513 +0000 UTC m=+259.907805381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gn8pw" (UID: "9dbad995-ab40-421a-bb52-4a6723f31ffd") : configmap "kube-rbac-proxy" not found Jul 17 18:37:22.004270 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:22.004268 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls podName:9dbad995-ab40-421a-bb52-4a6723f31ffd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:23.00426047 +0000 UTC m=+259.907834324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gn8pw" (UID: "9dbad995-ab40-421a-bb52-4a6723f31ffd") : secret "insights-runtime-extractor-tls" not found Jul 17 18:37:23.013556 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:23.013514 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:23.013556 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:23.013574 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:23.013993 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:23.013673 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:37:23.013993 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:23.013737 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm podName:9dbad995-ab40-421a-bb52-4a6723f31ffd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:25.01372178 +0000 UTC m=+261.917295634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gn8pw" (UID: "9dbad995-ab40-421a-bb52-4a6723f31ffd") : configmap "kube-rbac-proxy" not found Jul 17 18:37:23.013993 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:23.013678 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:37:23.013993 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:23.013821 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls podName:9dbad995-ab40-421a-bb52-4a6723f31ffd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:25.013808838 +0000 UTC m=+261.917382692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gn8pw" (UID: "9dbad995-ab40-421a-bb52-4a6723f31ffd") : secret "insights-runtime-extractor-tls" not found Jul 17 18:37:24.419722 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:24.419688 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tkrs6_f163c09b-fe4d-48de-9c56-7c8bbe212bc1/dns-node-resolver/0.log" Jul 17 18:37:24.525699 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:24.525662 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:24.525887 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:24.525712 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:24.525887 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:24.525755 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:24.525887 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:24.525833 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 17 18:37:24.525887 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:24.525834 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 17 18:37:24.525887 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:24.525858 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle podName:1c73c210-fc62-4860-93e4-c352a8021182 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:32.525839535 +0000 UTC m=+269.429413389 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle") pod "router-default-57c488db8b-cxf75" (UID: "1c73c210-fc62-4860-93e4-c352a8021182") : configmap references non-existent config key: service-ca.crt Jul 17 18:37:24.525887 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:24.525884 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert podName:d1af3565-93a3-4ce2-933d-c527fc2102ca nodeName:}" failed. No retries permitted until 2026-07-17 18:37:32.525873006 +0000 UTC m=+269.429446860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-ghl9n" (UID: "d1af3565-93a3-4ce2-933d-c527fc2102ca") : secret "networking-console-plugin-cert" not found Jul 17 18:37:24.526091 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:24.525896 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs podName:1c73c210-fc62-4860-93e4-c352a8021182 nodeName:}" failed. No retries permitted until 2026-07-17 18:37:32.525890099 +0000 UTC m=+269.429463952 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs") pod "router-default-57c488db8b-cxf75" (UID: "1c73c210-fc62-4860-93e4-c352a8021182") : secret "router-metrics-certs-default" not found Jul 17 18:37:25.030595 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:25.030539 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:25.030816 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:25.030615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:25.030816 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:25.030680 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:37:25.030816 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:25.030710 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:37:25.030816 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:25.030762 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm podName:9dbad995-ab40-421a-bb52-4a6723f31ffd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:29.030742657 +0000 UTC m=+265.934316513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gn8pw" (UID: "9dbad995-ab40-421a-bb52-4a6723f31ffd") : configmap "kube-rbac-proxy" not found Jul 17 18:37:25.030816 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:25.030780 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls podName:9dbad995-ab40-421a-bb52-4a6723f31ffd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:29.03077212 +0000 UTC m=+265.934345974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gn8pw" (UID: "9dbad995-ab40-421a-bb52-4a6723f31ffd") : secret "insights-runtime-extractor-tls" not found Jul 17 18:37:25.191964 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:25.191936 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qnnsp_d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c/node-ca/0.log" Jul 17 18:37:25.993041 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:25.993013 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kxtqg_9ec210e0-c89d-4835-b168-f3708c29d87f/serve-healthcheck-canary/0.log" Jul 17 18:37:26.796273 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:26.796240 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-mt27w_3ff9293e-8e04-4639-b2e8-4e806d121427/kube-storage-version-migrator-operator/0.log" Jul 17 18:37:29.062989 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:29.062944 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:29.063491 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:29.063090 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:29.063491 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:29.063094 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 17 18:37:29.063491 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:29.063131 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 17 18:37:29.063491 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:29.063195 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls podName:9dbad995-ab40-421a-bb52-4a6723f31ffd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:37.0631744 +0000 UTC m=+273.966748273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gn8pw" (UID: "9dbad995-ab40-421a-bb52-4a6723f31ffd") : secret "insights-runtime-extractor-tls" not found Jul 17 18:37:29.063491 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:29.063214 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm podName:9dbad995-ab40-421a-bb52-4a6723f31ffd nodeName:}" failed. No retries permitted until 2026-07-17 18:37:37.063204644 +0000 UTC m=+273.966778498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gn8pw" (UID: "9dbad995-ab40-421a-bb52-4a6723f31ffd") : configmap "kube-rbac-proxy" not found Jul 17 18:37:32.590859 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:32.590818 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:32.591272 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:32.590899 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:32.591272 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:32.590925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:32.591272 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:32.590992 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 17 18:37:32.591272 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:32.591062 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert podName:d1af3565-93a3-4ce2-933d-c527fc2102ca nodeName:}" failed. No retries permitted until 2026-07-17 18:37:48.591043412 +0000 UTC m=+285.494617267 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-ghl9n" (UID: "d1af3565-93a3-4ce2-933d-c527fc2102ca") : secret "networking-console-plugin-cert" not found Jul 17 18:37:32.591595 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:32.591575 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c73c210-fc62-4860-93e4-c352a8021182-service-ca-bundle\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:32.593551 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:32.593524 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c73c210-fc62-4860-93e4-c352a8021182-metrics-certs\") pod \"router-default-57c488db8b-cxf75\" (UID: \"1c73c210-fc62-4860-93e4-c352a8021182\") " pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:32.682614 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:32.682575 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:32.828162 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:32.828017 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57c488db8b-cxf75"] Jul 17 18:37:32.830934 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:37:32.830902 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c73c210_fc62_4860_93e4_c352a8021182.slice/crio-aa08076c3fd03f0b933b6583d5246e725add7fb6dcee76c10610c44e44d03896 WatchSource:0}: Error finding container aa08076c3fd03f0b933b6583d5246e725add7fb6dcee76c10610c44e44d03896: Status 404 returned error can't find the container with id aa08076c3fd03f0b933b6583d5246e725add7fb6dcee76c10610c44e44d03896 Jul 17 18:37:33.451385 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:33.451350 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57c488db8b-cxf75" event={"ID":"1c73c210-fc62-4860-93e4-c352a8021182","Type":"ContainerStarted","Data":"c61e89bd0683473cff905059c51d009d12ec64b8008af3d1853fa8bb6336f330"} Jul 17 18:37:33.451581 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:33.451408 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57c488db8b-cxf75" event={"ID":"1c73c210-fc62-4860-93e4-c352a8021182","Type":"ContainerStarted","Data":"aa08076c3fd03f0b933b6583d5246e725add7fb6dcee76c10610c44e44d03896"} Jul 17 18:37:33.473459 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:33.473378 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-57c488db8b-cxf75" podStartSLOduration=17.473359756 podStartE2EDuration="17.473359756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:33.472627363 +0000 UTC m=+270.376201238" watchObservedRunningTime="2026-07-17 18:37:33.473359756 +0000 UTC m=+270.376933611" Jul 17 18:37:33.683386 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:33.683351 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:33.686191 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:33.686162 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:34.455073 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:34.455040 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:34.456422 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:34.456398 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-57c488db8b-cxf75" Jul 17 18:37:37.129490 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:37.129433 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:37.129887 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:37.129505 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:37.129993 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:37.129971 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9dbad995-ab40-421a-bb52-4a6723f31ffd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:37.131991 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:37.131973 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9dbad995-ab40-421a-bb52-4a6723f31ffd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gn8pw\" (UID: \"9dbad995-ab40-421a-bb52-4a6723f31ffd\") " pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:37.298652 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:37.298599 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gn8pw" Jul 17 18:37:37.435781 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:37.435747 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gn8pw"] Jul 17 18:37:37.439969 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:37:37.439933 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbad995_ab40_421a_bb52_4a6723f31ffd.slice/crio-25041337f243577e3399b4988af4de8a68d69ddc2c8167645b6759abe027b720 WatchSource:0}: Error finding container 25041337f243577e3399b4988af4de8a68d69ddc2c8167645b6759abe027b720: Status 404 returned error can't find the container with id 25041337f243577e3399b4988af4de8a68d69ddc2c8167645b6759abe027b720 Jul 17 18:37:37.468423 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:37.468388 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gn8pw" event={"ID":"9dbad995-ab40-421a-bb52-4a6723f31ffd","Type":"ContainerStarted","Data":"25041337f243577e3399b4988af4de8a68d69ddc2c8167645b6759abe027b720"} Jul 17 18:37:43.493391 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:43.493353 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gn8pw" event={"ID":"9dbad995-ab40-421a-bb52-4a6723f31ffd","Type":"ContainerStarted","Data":"0ac5693e34365999eb50a27fb6a499b97bc503418c156cc36bbf3341199fafe1"} Jul 17 18:37:45.161508 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:45.161463 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" podUID="ee7b110a-cf7a-4a6a-91f0-d49900cf3769" Jul 17 18:37:45.501431 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:45.501339 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:37:46.506556 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:46.506472 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gn8pw" event={"ID":"9dbad995-ab40-421a-bb52-4a6723f31ffd","Type":"ContainerStarted","Data":"2b3488050ddd4a7a42790274b9f448bbc3d1f85ec976c1774e0b4d5a2e0aedbe"} Jul 17 18:37:48.625880 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.625839 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:48.626390 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.625938 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:37:48.628776 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.628750 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"image-registry-67cc6846cb-j9mzl\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:37:48.628920 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.628899 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d1af3565-93a3-4ce2-933d-c527fc2102ca-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-ghl9n\" (UID: \"d1af3565-93a3-4ce2-933d-c527fc2102ca\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:48.662676 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.662636 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" Jul 17 18:37:48.727094 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.727058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:37:48.729740 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.729711 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a782ebc-a204-4a31-a122-bd141b81b66d-metrics-tls\") pod \"dns-default-x2dl7\" (UID: \"8a782ebc-a204-4a31-a122-bd141b81b66d\") " pod="openshift-dns/dns-default-x2dl7" Jul 17 18:37:48.804297 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.804268 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h6x9z\"" Jul 17 18:37:48.812139 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.812102 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:37:48.818898 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.818871 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vz2rf\"" Jul 17 18:37:48.827030 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:48.826996 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x2dl7" Jul 17 18:37:49.951486 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:49.951458 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n"] Jul 17 18:37:49.954479 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:49.954430 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67cc6846cb-j9mzl"] Jul 17 18:37:49.957959 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:37:49.957930 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1af3565_93a3_4ce2_933d_c527fc2102ca.slice/crio-87c7d5532e22c5eeecbbb86393fbc6eb324f23c395f99029ecf8d2cbd0c44ce4 WatchSource:0}: Error finding container 87c7d5532e22c5eeecbbb86393fbc6eb324f23c395f99029ecf8d2cbd0c44ce4: Status 404 returned error can't find the container with id 87c7d5532e22c5eeecbbb86393fbc6eb324f23c395f99029ecf8d2cbd0c44ce4 Jul 17 18:37:49.958621 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:37:49.958567 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7b110a_cf7a_4a6a_91f0_d49900cf3769.slice/crio-2f14177aa33c154635f11d8ee7221a0122df87325a0584bfa025a0829058cc4b WatchSource:0}: Error finding container 2f14177aa33c154635f11d8ee7221a0122df87325a0584bfa025a0829058cc4b: Status 404 returned error can't find the container with id 2f14177aa33c154635f11d8ee7221a0122df87325a0584bfa025a0829058cc4b Jul 17 18:37:49.979501 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:49.979472 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x2dl7"] Jul 17 18:37:49.982173 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:37:49.982143 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a782ebc_a204_4a31_a122_bd141b81b66d.slice/crio-e1b13ec797341c11f60490a60f969b38f5075e08013b3f2b995b0b7ef886a40d WatchSource:0}: Error finding container e1b13ec797341c11f60490a60f969b38f5075e08013b3f2b995b0b7ef886a40d: Status 404 returned error can't find the container with id e1b13ec797341c11f60490a60f969b38f5075e08013b3f2b995b0b7ef886a40d Jul 17 18:37:50.522911 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:50.522872 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gn8pw" event={"ID":"9dbad995-ab40-421a-bb52-4a6723f31ffd","Type":"ContainerStarted","Data":"f08c00127431c77f2293227df3647f1d3d58675140c3096d9539190ad22e0375"} Jul 17 18:37:50.524398 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:50.524371 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" event={"ID":"ee7b110a-cf7a-4a6a-91f0-d49900cf3769","Type":"ContainerStarted","Data":"ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d"} Jul 17 18:37:50.524587 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:50.524419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" event={"ID":"ee7b110a-cf7a-4a6a-91f0-d49900cf3769","Type":"ContainerStarted","Data":"2f14177aa33c154635f11d8ee7221a0122df87325a0584bfa025a0829058cc4b"} Jul 17 18:37:50.524587 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:50.524468 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:37:50.525623 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:50.525597 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x2dl7" event={"ID":"8a782ebc-a204-4a31-a122-bd141b81b66d","Type":"ContainerStarted","Data":"e1b13ec797341c11f60490a60f969b38f5075e08013b3f2b995b0b7ef886a40d"} Jul 17 18:37:50.526634 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:50.526613 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" event={"ID":"d1af3565-93a3-4ce2-933d-c527fc2102ca","Type":"ContainerStarted","Data":"87c7d5532e22c5eeecbbb86393fbc6eb324f23c395f99029ecf8d2cbd0c44ce4"} Jul 17 18:37:50.552704 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:50.552657 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gn8pw" podStartSLOduration=17.298247655 podStartE2EDuration="29.552640086s" totalImagesPullingTime="12.254392431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 18:37:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:50.551898079 +0000 UTC m=+287.455471956" watchObservedRunningTime="2026-07-17 18:37:50.552640086 +0000 UTC m=+287.456213939" Jul 17 18:37:50.584682 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:50.584633 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" podStartSLOduration=286.584613418 podStartE2EDuration="4m46.584613418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:50.583404992 +0000 UTC m=+287.486978869" watchObservedRunningTime="2026-07-17 18:37:50.584613418 +0000 UTC m=+287.488187294" Jul 17 18:37:52.535796 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:52.535690 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x2dl7" event={"ID":"8a782ebc-a204-4a31-a122-bd141b81b66d","Type":"ContainerStarted","Data":"d803b3ba9fc48080aea797a9b3b44a690de90c8a5ccc4397b3da426d56ed8714"} Jul 17 18:37:52.535796 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:52.535747 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x2dl7" event={"ID":"8a782ebc-a204-4a31-a122-bd141b81b66d","Type":"ContainerStarted","Data":"08757eee1d7f7c38bc2697a17105d0768eb39cb47d46fcc3a181522b2d9ed72c"} Jul 17 18:37:52.535796 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:52.535785 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x2dl7" Jul 17 18:37:52.537295 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:52.537266 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" event={"ID":"d1af3565-93a3-4ce2-933d-c527fc2102ca","Type":"ContainerStarted","Data":"9f94d57cef99e55adf219395b00c3928f92abfef9dc355df71a3d4c985585a66"} Jul 17 18:37:52.556042 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:52.555993 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x2dl7" podStartSLOduration=252.337956882 podStartE2EDuration="4m14.55597605s" totalImagesPullingTime="2.218019168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:52.55503837 +0000 UTC m=+289.458612246" watchObservedRunningTime="2026-07-17 18:37:52.55597605 +0000 UTC m=+289.459549925" Jul 17 18:37:52.574210 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:52.574152 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-ghl9n" podStartSLOduration=34.953526488 podStartE2EDuration="36.574135302s" totalImagesPullingTime="1.620608814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:52.572890376 +0000 UTC m=+289.476464255" watchObservedRunningTime="2026-07-17 18:37:52.574135302 +0000 UTC m=+289.477709178" Jul 17 18:37:54.794305 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:54.794266 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l"] Jul 17 18:37:54.799933 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:54.799909 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l" Jul 17 18:37:54.802315 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:54.802290 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 17 18:37:54.802478 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:54.802380 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jv2tt\"" Jul 17 18:37:54.816772 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:54.816739 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l"] Jul 17 18:37:54.887026 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:54.886979 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e606a35c-d2e7-41c3-85eb-b24a1aa29e97-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-qf47l\" (UID: \"e606a35c-d2e7-41c3-85eb-b24a1aa29e97\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l" Jul 17 18:37:54.987858 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:54.987813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e606a35c-d2e7-41c3-85eb-b24a1aa29e97-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-qf47l\" (UID: \"e606a35c-d2e7-41c3-85eb-b24a1aa29e97\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l" Jul 17 18:37:54.990343 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:54.990310 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e606a35c-d2e7-41c3-85eb-b24a1aa29e97-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-qf47l\" (UID: \"e606a35c-d2e7-41c3-85eb-b24a1aa29e97\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l" Jul 17 18:37:55.109430 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:55.109334 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l" Jul 17 18:37:55.248668 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:55.248638 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l"] Jul 17 18:37:55.252137 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:37:55.252106 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode606a35c_d2e7_41c3_85eb_b24a1aa29e97.slice/crio-21acbcee14e14fe29eff208c0df4a57923a26fe903b2078ee28aeb0879d960a8 WatchSource:0}: Error finding container 21acbcee14e14fe29eff208c0df4a57923a26fe903b2078ee28aeb0879d960a8: Status 404 returned error can't find the container with id 21acbcee14e14fe29eff208c0df4a57923a26fe903b2078ee28aeb0879d960a8 Jul 17 18:37:55.548364 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:55.548331 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l" event={"ID":"e606a35c-d2e7-41c3-85eb-b24a1aa29e97","Type":"ContainerStarted","Data":"21acbcee14e14fe29eff208c0df4a57923a26fe903b2078ee28aeb0879d960a8"} Jul 17 18:37:57.556315 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.556274 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l" event={"ID":"e606a35c-d2e7-41c3-85eb-b24a1aa29e97","Type":"ContainerStarted","Data":"e838957cff03424b79b86f60e67c18a14a52eade2197a3dcf6032c8fcc253d3f"} Jul 17 18:37:57.556746 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.556410 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l" Jul 17 18:37:57.561225 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.561201 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l" Jul 17 18:37:57.576890 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.576838 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-qf47l" podStartSLOduration=2.310084129 podStartE2EDuration="3.576823473s" totalImagesPullingTime="1.266739344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:37:57.576600914 +0000 UTC m=+294.480174790" watchObservedRunningTime="2026-07-17 18:37:57.576823473 +0000 UTC m=+294.480397348" Jul 17 18:37:57.616769 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:37:57.616719 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6jqfb" podUID="7e6936f2-9870-44cc-abd2-99ae76f575ff" Jul 17 18:37:57.887405 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.887330 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-cx76z"] Jul 17 18:37:57.890742 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.890722 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:57.894875 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.894853 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 17 18:37:57.895501 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.895478 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 17 18:37:57.896781 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.896755 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 18:37:57.896936 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.896791 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6snhz\"" Jul 17 18:37:57.896936 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.896804 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 18:37:57.897419 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.897109 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 18:37:57.900844 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:57.900819 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-cx76z"] Jul 17 18:37:58.012377 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.012333 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh9v2\" (UniqueName: \"kubernetes.io/projected/d2dfa1cb-2b67-4ba2-b271-30f06b605820-kube-api-access-rh9v2\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.012591 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.012407 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfa1cb-2b67-4ba2-b271-30f06b605820-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.012591 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.012500 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2dfa1cb-2b67-4ba2-b271-30f06b605820-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.012591 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.012537 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2dfa1cb-2b67-4ba2-b271-30f06b605820-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.113740 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.113698 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfa1cb-2b67-4ba2-b271-30f06b605820-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.113938 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.113746 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2dfa1cb-2b67-4ba2-b271-30f06b605820-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.113938 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.113773 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2dfa1cb-2b67-4ba2-b271-30f06b605820-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.113938 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.113819 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh9v2\" (UniqueName: \"kubernetes.io/projected/d2dfa1cb-2b67-4ba2-b271-30f06b605820-kube-api-access-rh9v2\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.114584 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.114563 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2dfa1cb-2b67-4ba2-b271-30f06b605820-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.116501 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.116477 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d2dfa1cb-2b67-4ba2-b271-30f06b605820-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.116609 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.116501 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d2dfa1cb-2b67-4ba2-b271-30f06b605820-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.126392 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.126366 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh9v2\" (UniqueName: \"kubernetes.io/projected/d2dfa1cb-2b67-4ba2-b271-30f06b605820-kube-api-access-rh9v2\") pod \"prometheus-operator-5945b5cfd-cx76z\" (UID: \"d2dfa1cb-2b67-4ba2-b271-30f06b605820\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.200053 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.200019 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" Jul 17 18:37:58.340800 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.340765 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-cx76z"] Jul 17 18:37:58.343817 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:37:58.343793 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2dfa1cb_2b67_4ba2_b271_30f06b605820.slice/crio-56bffbb74b397f8b81b35af190e03317028048aa321d50e6e6f7784a535668c5 WatchSource:0}: Error finding container 56bffbb74b397f8b81b35af190e03317028048aa321d50e6e6f7784a535668c5: Status 404 returned error can't find the container with id 56bffbb74b397f8b81b35af190e03317028048aa321d50e6e6f7784a535668c5 Jul 17 18:37:58.561226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:37:58.561140 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" event={"ID":"d2dfa1cb-2b67-4ba2-b271-30f06b605820","Type":"ContainerStarted","Data":"56bffbb74b397f8b81b35af190e03317028048aa321d50e6e6f7784a535668c5"} Jul 17 18:38:00.570600 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:00.570562 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" event={"ID":"d2dfa1cb-2b67-4ba2-b271-30f06b605820","Type":"ContainerStarted","Data":"8086d35768ae8405787cbac415c00ad337a9eca8594a20b6ad10e1c83b3424ce"} Jul 17 18:38:00.571023 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:00.570625 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" event={"ID":"d2dfa1cb-2b67-4ba2-b271-30f06b605820","Type":"ContainerStarted","Data":"78ee284084e8c6cdd78424afa4b1b6f849710bc5485b6ad43d2933fedf734d2a"} Jul 17 18:38:00.601271 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:00.601212 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-cx76z" podStartSLOduration=1.909347233 podStartE2EDuration="3.601193543s" totalImagesPullingTime="1.69184631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:00.600137269 +0000 UTC m=+297.503711147" watchObservedRunningTime="2026-07-17 18:38:00.601193543 +0000 UTC m=+297.504767421" Jul 17 18:38:02.514858 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.514825 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg"] Jul 17 18:38:02.518363 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.518337 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.521179 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.521150 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qqckc\"" Jul 17 18:38:02.521179 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.521183 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 18:38:02.521368 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.521223 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 18:38:02.526968 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.526943 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vvrnl"] Jul 17 18:38:02.530040 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.530014 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg"] Jul 17 18:38:02.530156 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.530127 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.532530 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.532506 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 18:38:02.532670 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.532512 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 18:38:02.532670 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.532552 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bkwv8\"" Jul 17 18:38:02.532936 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.532918 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 18:38:02.542331 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.542303 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x2dl7" Jul 17 18:38:02.651363 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651326 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.651363 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-wtmp\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.651634 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651399 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-tls\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.651634 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.651742 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651640 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2s84\" (UniqueName: \"kubernetes.io/projected/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-kube-api-access-q2s84\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.651742 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651693 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6860e99a-1e8f-4a35-8ada-18e31b298764-metrics-client-ca\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.651742 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651736 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6860e99a-1e8f-4a35-8ada-18e31b298764-root\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.651891 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651787 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.651891 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651834 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.651891 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651861 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwlhd\" (UniqueName: \"kubernetes.io/projected/6860e99a-1e8f-4a35-8ada-18e31b298764-kube-api-access-bwlhd\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.652034 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651894 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-textfile\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.652034 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651929 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6860e99a-1e8f-4a35-8ada-18e31b298764-sys\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.652034 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.651953 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.752739 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.752705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6860e99a-1e8f-4a35-8ada-18e31b298764-root\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.752739 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.752743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.752992 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.752775 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.752992 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.752792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwlhd\" (UniqueName: \"kubernetes.io/projected/6860e99a-1e8f-4a35-8ada-18e31b298764-kube-api-access-bwlhd\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.752992 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.752812 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-textfile\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.752992 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.752823 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6860e99a-1e8f-4a35-8ada-18e31b298764-root\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.752992 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.752835 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6860e99a-1e8f-4a35-8ada-18e31b298764-sys\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.752992 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.752878 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6860e99a-1e8f-4a35-8ada-18e31b298764-sys\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.752992 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.752910 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.752992 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.752983 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.753368 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.753034 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-wtmp\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.753368 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.753068 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-tls\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.753368 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.753103 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.753368 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.753127 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2s84\" (UniqueName: \"kubernetes.io/projected/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-kube-api-access-q2s84\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.753368 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:38:02.753189 2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 18:38:02.753368 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.753191 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-textfile\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.753937 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.753908 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-wtmp\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.754131 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.754021 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.754241 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:38:02.754170 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-tls podName:6860e99a-1e8f-4a35-8ada-18e31b298764 nodeName:}" failed. No retries permitted until 2026-07-17 18:38:03.253240374 +0000 UTC m=+300.156814234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-tls") pod "node-exporter-vvrnl" (UID: "6860e99a-1e8f-4a35-8ada-18e31b298764") : secret "node-exporter-tls" not found Jul 17 18:38:02.754364 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.754349 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6860e99a-1e8f-4a35-8ada-18e31b298764-metrics-client-ca\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.754501 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.754479 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.755102 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.755072 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6860e99a-1e8f-4a35-8ada-18e31b298764-metrics-client-ca\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.756536 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.756515 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.758026 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.757992 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.758116 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.757999 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.762536 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.762511 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwlhd\" (UniqueName: \"kubernetes.io/projected/6860e99a-1e8f-4a35-8ada-18e31b298764-kube-api-access-bwlhd\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:02.763163 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.763142 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2s84\" (UniqueName: \"kubernetes.io/projected/0b075b8a-e5b0-4008-ae0e-c2a383482e5d-kube-api-access-q2s84\") pod \"openshift-state-metrics-79f6958cdf-kjqtg\" (UID: \"0b075b8a-e5b0-4008-ae0e-c2a383482e5d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.827942 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.827855 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" Jul 17 18:38:02.971898 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:02.971865 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg"] Jul 17 18:38:02.974380 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:38:02.974346 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b075b8a_e5b0_4008_ae0e_c2a383482e5d.slice/crio-9d187d48a5e5246cc681cb7b0d742e16bd8fc80ef5082f9aede148f4671534df WatchSource:0}: Error finding container 9d187d48a5e5246cc681cb7b0d742e16bd8fc80ef5082f9aede148f4671534df: Status 404 returned error can't find the container with id 9d187d48a5e5246cc681cb7b0d742e16bd8fc80ef5082f9aede148f4671534df Jul 17 18:38:03.260480 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:03.260406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-tls\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:03.262930 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:03.262903 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6860e99a-1e8f-4a35-8ada-18e31b298764-node-exporter-tls\") pod \"node-exporter-vvrnl\" (UID: \"6860e99a-1e8f-4a35-8ada-18e31b298764\") " pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:03.440020 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:03.439989 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvrnl" Jul 17 18:38:03.451027 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:38:03.450998 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6860e99a_1e8f_4a35_8ada_18e31b298764.slice/crio-ab54cc555a17a8c1856c6b425be5084994ff5952ceeb3b4e777e5303c535d0b4 WatchSource:0}: Error finding container ab54cc555a17a8c1856c6b425be5084994ff5952ceeb3b4e777e5303c535d0b4: Status 404 returned error can't find the container with id ab54cc555a17a8c1856c6b425be5084994ff5952ceeb3b4e777e5303c535d0b4 Jul 17 18:38:03.501169 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:03.501144 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 18:38:03.581777 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:03.581617 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" event={"ID":"0b075b8a-e5b0-4008-ae0e-c2a383482e5d","Type":"ContainerStarted","Data":"0a4c3d3785917f74caf56a72e43d17207ca4ee131238ab849734af1c988c4e32"} Jul 17 18:38:03.581777 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:03.581676 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" event={"ID":"0b075b8a-e5b0-4008-ae0e-c2a383482e5d","Type":"ContainerStarted","Data":"353de183a8a5034b2a3f82d44d62d393bd6cb37cddeb41a8333873121557d53a"} Jul 17 18:38:03.581777 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:03.581694 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" event={"ID":"0b075b8a-e5b0-4008-ae0e-c2a383482e5d","Type":"ContainerStarted","Data":"9d187d48a5e5246cc681cb7b0d742e16bd8fc80ef5082f9aede148f4671534df"} Jul 17 18:38:03.605711 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:03.582702 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvrnl" event={"ID":"6860e99a-1e8f-4a35-8ada-18e31b298764","Type":"ContainerStarted","Data":"ab54cc555a17a8c1856c6b425be5084994ff5952ceeb3b4e777e5303c535d0b4"} Jul 17 18:38:04.587678 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:04.587584 2563 generic.go:356] "Generic (PLEG): container finished" podID="e706fba2-dcfe-4886-aa60-2edb19cd22ee" containerID="988866a50bc0bea64983d4e9fbe0e18786a4a5a7b1f402779d9de3bd77cfabe4" exitCode=1 Jul 17 18:38:04.587678 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:04.587635 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" event={"ID":"e706fba2-dcfe-4886-aa60-2edb19cd22ee","Type":"ContainerDied","Data":"988866a50bc0bea64983d4e9fbe0e18786a4a5a7b1f402779d9de3bd77cfabe4"} Jul 17 18:38:04.587678 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:04.587682 2563 scope.go:117] "RemoveContainer" containerID="7dd3700685cb881f67caa18b1e14164a259ed6c590504574fbafd98cd406085a" Jul 17 18:38:04.588244 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:04.588054 2563 scope.go:117] "RemoveContainer" containerID="988866a50bc0bea64983d4e9fbe0e18786a4a5a7b1f402779d9de3bd77cfabe4" Jul 17 18:38:04.588244 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:38:04.588230 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-596ddd7bc4-f7n24_open-cluster-management-agent-addon(e706fba2-dcfe-4886-aa60-2edb19cd22ee)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" podUID="e706fba2-dcfe-4886-aa60-2edb19cd22ee" Jul 17 18:38:05.219131 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.219089 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:38:05.481496 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.481396 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7475d87648-pf47s"] Jul 17 18:38:05.485107 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.485085 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.492516 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.492497 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 18:38:05.492956 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.492937 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-89hr2s19egsbf\"" Jul 17 18:38:05.493114 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.493088 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tblsw\"" Jul 17 18:38:05.493223 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.493210 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 18:38:05.493429 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.493408 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 18:38:05.493529 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.493491 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 18:38:05.493655 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.493641 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 18:38:05.502218 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.502192 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7475d87648-pf47s"] Jul 17 18:38:05.579914 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.579868 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.579914 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.579917 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.580128 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.579937 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-metrics-client-ca\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.580128 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.580018 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-tls\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.580128 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.580053 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-grpc-tls\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.580128 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.580102 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.580128 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.580130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.580333 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.580149 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh7zc\" (UniqueName: \"kubernetes.io/projected/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-kube-api-access-dh7zc\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.592581 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.592541 2563 generic.go:356] "Generic (PLEG): container finished" podID="6860e99a-1e8f-4a35-8ada-18e31b298764" containerID="cb5622e72c17798fdee7a78bcaf5b4896b65d9814f38c84d8f0d3c5dd09f1382" exitCode=0 Jul 17 18:38:05.593048 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.592626 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvrnl" event={"ID":"6860e99a-1e8f-4a35-8ada-18e31b298764","Type":"ContainerDied","Data":"cb5622e72c17798fdee7a78bcaf5b4896b65d9814f38c84d8f0d3c5dd09f1382"} Jul 17 18:38:05.594470 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.594429 2563 scope.go:117] "RemoveContainer" containerID="988866a50bc0bea64983d4e9fbe0e18786a4a5a7b1f402779d9de3bd77cfabe4" Jul 17 18:38:05.594651 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:38:05.594630 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-596ddd7bc4-f7n24_open-cluster-management-agent-addon(e706fba2-dcfe-4886-aa60-2edb19cd22ee)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" podUID="e706fba2-dcfe-4886-aa60-2edb19cd22ee" Jul 17 18:38:05.596400 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.596372 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" event={"ID":"0b075b8a-e5b0-4008-ae0e-c2a383482e5d","Type":"ContainerStarted","Data":"9cf36fe94757a26fa816ba0f44f8787c6420d580966c9cd76b7186227ff25e17"} Jul 17 18:38:05.636786 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.636735 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-kjqtg" podStartSLOduration=1.874577327 podStartE2EDuration="3.636716842s" totalImagesPullingTime="1.762139515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:05.635937022 +0000 UTC m=+302.539510897" watchObservedRunningTime="2026-07-17 18:38:05.636716842 +0000 UTC m=+302.540290771" Jul 17 18:38:05.680821 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.680791 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-grpc-tls\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.680984 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.680845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.680984 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.680874 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.680984 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.680891 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh7zc\" (UniqueName: \"kubernetes.io/projected/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-kube-api-access-dh7zc\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.680984 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.680954 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.681262 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.680989 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.681262 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.681025 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-metrics-client-ca\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.681262 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.681064 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-tls\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.682830 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.682742 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-metrics-client-ca\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.684031 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.684001 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-grpc-tls\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.684240 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.684188 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.684384 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.684363 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.684633 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.684610 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.684866 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.684843 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-tls\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.685607 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.685584 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.702464 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.702413 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh7zc\" (UniqueName: \"kubernetes.io/projected/3d5c6871-9db6-451d-bcb3-ddc377c9d37b-kube-api-access-dh7zc\") pod \"thanos-querier-7475d87648-pf47s\" (UID: \"3d5c6871-9db6-451d-bcb3-ddc377c9d37b\") " pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.794555 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.794427 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:05.940124 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.940090 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7475d87648-pf47s"] Jul 17 18:38:05.943285 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:38:05.943255 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5c6871_9db6_451d_bcb3_ddc377c9d37b.slice/crio-20900da1b07ccadadce848afe49269e714e8283bc8275b84498295cb4bf62cb6 WatchSource:0}: Error finding container 20900da1b07ccadadce848afe49269e714e8283bc8275b84498295cb4bf62cb6: Status 404 returned error can't find the container with id 20900da1b07ccadadce848afe49269e714e8283bc8275b84498295cb4bf62cb6 Jul 17 18:38:05.945380 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:05.945363 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:38:06.602146 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.602107 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" event={"ID":"3d5c6871-9db6-451d-bcb3-ddc377c9d37b","Type":"ContainerStarted","Data":"20900da1b07ccadadce848afe49269e714e8283bc8275b84498295cb4bf62cb6"} Jul 17 18:38:06.604715 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.604680 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvrnl" event={"ID":"6860e99a-1e8f-4a35-8ada-18e31b298764","Type":"ContainerStarted","Data":"b0ebe29badd1d042c1fb5f836d83841440b73d51ec7a33f91aed26ae08e2f621"} Jul 17 18:38:06.604883 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.604751 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvrnl" event={"ID":"6860e99a-1e8f-4a35-8ada-18e31b298764","Type":"ContainerStarted","Data":"1cf100c4e9444fa2d514312991b7053ce8bb502f6c9ce20667d8c6b8caa273b4"} Jul 17 18:38:06.633756 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.633690 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vvrnl" podStartSLOduration=3.152655529 podStartE2EDuration="4.633667808s" totalImagesPullingTime="1.403162585s" totalInitContainerRuntime="77.849694ms" isStatefulPod=true podCreationTimestamp="2026-07-17 18:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:06.632632012 +0000 UTC m=+303.536205901" watchObservedRunningTime="2026-07-17 18:38:06.633667808 +0000 UTC m=+303.537241685" Jul 17 18:38:06.817520 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.817437 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-75c8cfd894-7jmvl"] Jul 17 18:38:06.820821 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.820796 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:06.826490 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.823938 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 18:38:06.826490 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.824180 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 18:38:06.826490 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.824656 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hjqsf\"" Jul 17 18:38:06.826490 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.825027 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 18:38:06.826490 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.825820 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2q1u6kgaeg30a\"" Jul 17 18:38:06.826490 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.825908 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 18:38:06.833380 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.833350 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75c8cfd894-7jmvl"] Jul 17 18:38:06.995254 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.995221 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/02db1c12-fbff-4d59-a0ae-c0935d4694a0-secret-metrics-server-client-certs\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:06.995439 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.995260 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/02db1c12-fbff-4d59-a0ae-c0935d4694a0-secret-metrics-server-tls\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:06.995439 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.995289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xffpx\" (UniqueName: \"kubernetes.io/projected/02db1c12-fbff-4d59-a0ae-c0935d4694a0-kube-api-access-xffpx\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:06.995439 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.995390 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/02db1c12-fbff-4d59-a0ae-c0935d4694a0-audit-log\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:06.995439 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.995431 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/02db1c12-fbff-4d59-a0ae-c0935d4694a0-client-ca-bundle\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:06.995633 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.995517 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/02db1c12-fbff-4d59-a0ae-c0935d4694a0-metrics-server-audit-profiles\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:06.995633 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:06.995537 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02db1c12-fbff-4d59-a0ae-c0935d4694a0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.096288 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.096249 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/02db1c12-fbff-4d59-a0ae-c0935d4694a0-secret-metrics-server-client-certs\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.096288 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.096293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/02db1c12-fbff-4d59-a0ae-c0935d4694a0-secret-metrics-server-tls\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.096572 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.096316 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xffpx\" (UniqueName: \"kubernetes.io/projected/02db1c12-fbff-4d59-a0ae-c0935d4694a0-kube-api-access-xffpx\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.096572 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.096369 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/02db1c12-fbff-4d59-a0ae-c0935d4694a0-audit-log\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.096572 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.096410 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/02db1c12-fbff-4d59-a0ae-c0935d4694a0-client-ca-bundle\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.096572 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.096508 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/02db1c12-fbff-4d59-a0ae-c0935d4694a0-metrics-server-audit-profiles\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.096572 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.096533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02db1c12-fbff-4d59-a0ae-c0935d4694a0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.096911 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.096879 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/02db1c12-fbff-4d59-a0ae-c0935d4694a0-audit-log\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.097366 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.097343 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02db1c12-fbff-4d59-a0ae-c0935d4694a0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.097649 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.097633 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/02db1c12-fbff-4d59-a0ae-c0935d4694a0-metrics-server-audit-profiles\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.099756 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.099710 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/02db1c12-fbff-4d59-a0ae-c0935d4694a0-secret-metrics-server-tls\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.100621 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.100595 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/02db1c12-fbff-4d59-a0ae-c0935d4694a0-client-ca-bundle\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.100765 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.100742 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/02db1c12-fbff-4d59-a0ae-c0935d4694a0-secret-metrics-server-client-certs\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.105812 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.105787 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xffpx\" (UniqueName: \"kubernetes.io/projected/02db1c12-fbff-4d59-a0ae-c0935d4694a0-kube-api-access-xffpx\") pod \"metrics-server-75c8cfd894-7jmvl\" (UID: \"02db1c12-fbff-4d59-a0ae-c0935d4694a0\") " pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.136633 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.136594 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:07.147496 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.147424 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc"] Jul 17 18:38:07.155072 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.155039 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:07.159020 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.158986 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 18:38:07.159190 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.159115 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7r8ht\"" Jul 17 18:38:07.163616 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.163540 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc"] Jul 17 18:38:07.298133 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.298044 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869sm\" (UniqueName: \"kubernetes.io/projected/c86ee112-e81f-4e44-bb11-a04adea1c35b-kube-api-access-869sm\") pod \"monitoring-plugin-7b7b7bc8bf-f58fc\" (UID: \"c86ee112-e81f-4e44-bb11-a04adea1c35b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:07.298287 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.298179 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c86ee112-e81f-4e44-bb11-a04adea1c35b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-f58fc\" (UID: \"c86ee112-e81f-4e44-bb11-a04adea1c35b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:07.310081 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.310007 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75c8cfd894-7jmvl"] Jul 17 18:38:07.312598 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:38:07.312563 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02db1c12_fbff_4d59_a0ae_c0935d4694a0.slice/crio-52c13bbf80a4eca0c4f6f9e93da5ce317ad47972edfac8aa2c28152e2601655a WatchSource:0}: Error finding container 52c13bbf80a4eca0c4f6f9e93da5ce317ad47972edfac8aa2c28152e2601655a: Status 404 returned error can't find the container with id 52c13bbf80a4eca0c4f6f9e93da5ce317ad47972edfac8aa2c28152e2601655a Jul 17 18:38:07.399338 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.399288 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c86ee112-e81f-4e44-bb11-a04adea1c35b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-f58fc\" (UID: \"c86ee112-e81f-4e44-bb11-a04adea1c35b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:07.399576 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.399380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869sm\" (UniqueName: \"kubernetes.io/projected/c86ee112-e81f-4e44-bb11-a04adea1c35b-kube-api-access-869sm\") pod \"monitoring-plugin-7b7b7bc8bf-f58fc\" (UID: \"c86ee112-e81f-4e44-bb11-a04adea1c35b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:07.399576 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:38:07.399468 2563 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 17 18:38:07.399576 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:38:07.399546 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c86ee112-e81f-4e44-bb11-a04adea1c35b-monitoring-plugin-cert podName:c86ee112-e81f-4e44-bb11-a04adea1c35b nodeName:}" failed. No retries permitted until 2026-07-17 18:38:07.899525522 +0000 UTC m=+304.803099389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c86ee112-e81f-4e44-bb11-a04adea1c35b-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-f58fc" (UID: "c86ee112-e81f-4e44-bb11-a04adea1c35b") : secret "monitoring-plugin-cert" not found Jul 17 18:38:07.408142 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.408112 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-869sm\" (UniqueName: \"kubernetes.io/projected/c86ee112-e81f-4e44-bb11-a04adea1c35b-kube-api-access-869sm\") pod \"monitoring-plugin-7b7b7bc8bf-f58fc\" (UID: \"c86ee112-e81f-4e44-bb11-a04adea1c35b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:07.609425 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.609326 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" event={"ID":"02db1c12-fbff-4d59-a0ae-c0935d4694a0","Type":"ContainerStarted","Data":"52c13bbf80a4eca0c4f6f9e93da5ce317ad47972edfac8aa2c28152e2601655a"} Jul 17 18:38:07.903688 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.903641 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c86ee112-e81f-4e44-bb11-a04adea1c35b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-f58fc\" (UID: \"c86ee112-e81f-4e44-bb11-a04adea1c35b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:07.906674 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:07.906643 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c86ee112-e81f-4e44-bb11-a04adea1c35b-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-f58fc\" (UID: \"c86ee112-e81f-4e44-bb11-a04adea1c35b\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:08.077823 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.077789 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:08.262460 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.262083 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bdf799767-jmn7l"] Jul 17 18:38:08.265689 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.265662 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.269280 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.269246 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 18:38:08.271393 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.271365 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 18:38:08.271561 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.271439 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nt4s4\"" Jul 17 18:38:08.271662 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.271636 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 18:38:08.271662 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.271661 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 18:38:08.271841 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.271690 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 18:38:08.272378 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.272361 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 18:38:08.273337 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.273185 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 18:38:08.276831 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.276803 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 18:38:08.282270 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.282232 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdf799767-jmn7l"] Jul 17 18:38:08.408364 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.408330 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-config\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.408596 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.408376 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq4lx\" (UniqueName: \"kubernetes.io/projected/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-kube-api-access-dq4lx\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.408596 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.408404 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-service-ca\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.408596 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.408422 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-trusted-ca-bundle\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.408596 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.408554 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-oauth-serving-cert\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.408802 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.408597 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-serving-cert\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.408802 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.408688 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-oauth-config\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.511106 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.510011 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-serving-cert\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.511106 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.510071 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-oauth-config\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.511106 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.510165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-config\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.511106 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.510207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq4lx\" (UniqueName: \"kubernetes.io/projected/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-kube-api-access-dq4lx\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.511106 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.510268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-service-ca\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.511106 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.510294 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-trusted-ca-bundle\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.511106 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.510326 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-oauth-serving-cert\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.512231 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.511727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-oauth-serving-cert\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.512231 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.511910 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-config\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.512231 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.511964 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-service-ca\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.512678 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.512440 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-trusted-ca-bundle\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.520118 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.519888 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-oauth-config\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.520654 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.520606 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-serving-cert\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.529415 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.529351 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq4lx\" (UniqueName: \"kubernetes.io/projected/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-kube-api-access-dq4lx\") pod \"console-6bdf799767-jmn7l\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.543354 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.543312 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc"] Jul 17 18:38:08.550270 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:38:08.550234 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86ee112_e81f_4e44_bb11_a04adea1c35b.slice/crio-017778186fbb38fe71ee1acee62683f8fb270402fd599bc9854081ca718f423c WatchSource:0}: Error finding container 017778186fbb38fe71ee1acee62683f8fb270402fd599bc9854081ca718f423c: Status 404 returned error can't find the container with id 017778186fbb38fe71ee1acee62683f8fb270402fd599bc9854081ca718f423c Jul 17 18:38:08.580854 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.580630 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:08.619240 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.619177 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" event={"ID":"c86ee112-e81f-4e44-bb11-a04adea1c35b","Type":"ContainerStarted","Data":"017778186fbb38fe71ee1acee62683f8fb270402fd599bc9854081ca718f423c"} Jul 17 18:38:08.621311 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.621277 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" event={"ID":"3d5c6871-9db6-451d-bcb3-ddc377c9d37b","Type":"ContainerStarted","Data":"cc273b96b18cf14fa89cda0b600e87607c9b85c555fb0e8513b04e14f6659712"} Jul 17 18:38:08.691083 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.691046 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67cc6846cb-j9mzl"] Jul 17 18:38:08.696249 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.696117 2563 patch_prober.go:28] interesting pod/image-registry-67cc6846cb-j9mzl container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 17 18:38:08.696249 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.696178 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" podUID="ee7b110a-cf7a-4a6a-91f0-d49900cf3769" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 18:38:08.769008 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:08.768977 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bdf799767-jmn7l"] Jul 17 18:38:09.171836 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:09.171809 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:38:09.172337 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:09.172319 2563 scope.go:117] "RemoveContainer" containerID="988866a50bc0bea64983d4e9fbe0e18786a4a5a7b1f402779d9de3bd77cfabe4" Jul 17 18:38:09.172608 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:38:09.172569 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-596ddd7bc4-f7n24_open-cluster-management-agent-addon(e706fba2-dcfe-4886-aa60-2edb19cd22ee)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" podUID="e706fba2-dcfe-4886-aa60-2edb19cd22ee" Jul 17 18:38:09.626652 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:09.626338 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" event={"ID":"02db1c12-fbff-4d59-a0ae-c0935d4694a0","Type":"ContainerStarted","Data":"77fae8f8f2706384b20836505bc034b423509eede94b369ef543871940ac2dcc"} Jul 17 18:38:09.631241 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:09.631207 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" event={"ID":"3d5c6871-9db6-451d-bcb3-ddc377c9d37b","Type":"ContainerStarted","Data":"954caffdc02e65e473e8ced9195eccea44f40c0d58119f3561e90dc8894e0cbb"} Jul 17 18:38:09.631544 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:09.631290 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" event={"ID":"3d5c6871-9db6-451d-bcb3-ddc377c9d37b","Type":"ContainerStarted","Data":"657a45b20d080311092ff65092f80bf2cce4899c7d6f7b5a16ced87d750a754d"} Jul 17 18:38:09.632742 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:09.632677 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdf799767-jmn7l" event={"ID":"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f","Type":"ContainerStarted","Data":"5f820a75eac98c49567487279f2dbb47c8f770b34608e9a32dcfe7c732a45c4c"} Jul 17 18:38:09.652368 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:09.652303 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" podStartSLOduration=1.795968386 podStartE2EDuration="3.652280283s" totalImagesPullingTime="1.856311897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:09.649870863 +0000 UTC m=+306.553444756" watchObservedRunningTime="2026-07-17 18:38:09.652280283 +0000 UTC m=+306.555854178" Jul 17 18:38:10.617231 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:10.617120 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:38:11.646309 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:11.646220 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" event={"ID":"3d5c6871-9db6-451d-bcb3-ddc377c9d37b","Type":"ContainerStarted","Data":"32650e6675b082d97604fbc3ddf6662910abf0a6102881e48f6158d6174a8694"} Jul 17 18:38:11.646309 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:11.646308 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" event={"ID":"3d5c6871-9db6-451d-bcb3-ddc377c9d37b","Type":"ContainerStarted","Data":"e6a436d2e1f63a992e85336e299f0ee4bfa104ebb7722dd3470646399b4aa9ca"} Jul 17 18:38:11.646828 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:11.646362 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:11.646828 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:11.646401 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" event={"ID":"3d5c6871-9db6-451d-bcb3-ddc377c9d37b","Type":"ContainerStarted","Data":"3865e383bae3f3688af38ed4f3ff95a67b591f659220f8e071b9b5ec16420ac0"} Jul 17 18:38:11.648067 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:11.648041 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" event={"ID":"c86ee112-e81f-4e44-bb11-a04adea1c35b","Type":"ContainerStarted","Data":"c73ff8287ad27358e342f5abd2047b47e329845992a349db019657058b824eb4"} Jul 17 18:38:11.648293 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:11.648273 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:11.655081 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:11.655050 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" Jul 17 18:38:11.687898 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:11.687835 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" podStartSLOduration=2.654064804 podStartE2EDuration="6.687816134s" totalImagesPullingTime="4.03375133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:38:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:11.684383107 +0000 UTC m=+308.587956984" watchObservedRunningTime="2026-07-17 18:38:11.687816134 +0000 UTC m=+308.591390009" Jul 17 18:38:12.655696 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:12.655661 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdf799767-jmn7l" event={"ID":"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f","Type":"ContainerStarted","Data":"1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f"} Jul 17 18:38:12.675990 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:12.675890 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-f58fc" podStartSLOduration=3.505821742 podStartE2EDuration="5.67587286s" totalImagesPullingTime="2.170051118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:38:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:11.717884601 +0000 UTC m=+308.621458479" watchObservedRunningTime="2026-07-17 18:38:12.67587286 +0000 UTC m=+309.579446735" Jul 17 18:38:12.676138 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:12.676122 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bdf799767-jmn7l" podStartSLOduration=1.368716797 podStartE2EDuration="4.676117839s" totalImagesPullingTime="3.307401042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:38:12.675003028 +0000 UTC m=+309.578576905" watchObservedRunningTime="2026-07-17 18:38:12.676117839 +0000 UTC m=+309.579691714" Jul 17 18:38:17.662489 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:17.662437 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7475d87648-pf47s" Jul 17 18:38:18.581074 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:18.581027 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:18.581074 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:18.581087 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:18.586788 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:18.586758 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:18.684359 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:18.684325 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:38:18.696770 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:18.696740 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:38:20.617191 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:20.617158 2563 scope.go:117] "RemoveContainer" containerID="988866a50bc0bea64983d4e9fbe0e18786a4a5a7b1f402779d9de3bd77cfabe4" Jul 17 18:38:21.693558 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:21.693518 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" event={"ID":"e706fba2-dcfe-4886-aa60-2edb19cd22ee","Type":"ContainerStarted","Data":"0d8b56669da258836af12f3528f5df4347f64de1f00b52d251991b2b4bd6628a"} Jul 17 18:38:21.694018 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:21.693799 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:38:21.695523 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:21.695503 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-596ddd7bc4-f7n24" Jul 17 18:38:27.137560 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:27.137522 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:27.137969 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:27.137592 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:31.819642 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:31.819605 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57c488db8b-cxf75_1c73c210-fc62-4860-93e4-c352a8021182/router/0.log" Jul 17 18:38:31.838734 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:31.838700 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kxtqg_9ec210e0-c89d-4835-b168-f3708c29d87f/serve-healthcheck-canary/0.log" Jul 17 18:38:33.723000 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:33.722952 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" podUID="ee7b110a-cf7a-4a6a-91f0-d49900cf3769" containerName="registry" containerID="cri-o://ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d" gracePeriod=30 Jul 17 18:38:33.961226 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:33.961198 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:38:34.039051 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.038950 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-trusted-ca\") pod \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " Jul 17 18:38:34.039051 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.039022 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-bound-sa-token\") pod \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " Jul 17 18:38:34.039273 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.039060 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-ca-trust-extracted\") pod \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " Jul 17 18:38:34.039273 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.039094 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xgvh\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-kube-api-access-5xgvh\") pod \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " Jul 17 18:38:34.039273 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.039126 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-installation-pull-secrets\") pod \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " Jul 17 18:38:34.039273 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.039169 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-image-registry-private-configuration\") pod \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " Jul 17 18:38:34.039273 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.039204 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") pod \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " Jul 17 18:38:34.039273 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.039253 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-certificates\") pod \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\" (UID: \"ee7b110a-cf7a-4a6a-91f0-d49900cf3769\") " Jul 17 18:38:34.039604 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.039390 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ee7b110a-cf7a-4a6a-91f0-d49900cf3769" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:38:34.039604 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.039545 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-trusted-ca\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:38:34.039798 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.039778 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ee7b110a-cf7a-4a6a-91f0-d49900cf3769" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:38:34.042210 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.042175 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ee7b110a-cf7a-4a6a-91f0-d49900cf3769" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:38:34.042336 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.042244 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ee7b110a-cf7a-4a6a-91f0-d49900cf3769" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:38:34.042336 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.042310 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ee7b110a-cf7a-4a6a-91f0-d49900cf3769" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:38:34.042434 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.042340 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ee7b110a-cf7a-4a6a-91f0-d49900cf3769" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:38:34.042434 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.042363 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-kube-api-access-5xgvh" (OuterVolumeSpecName: "kube-api-access-5xgvh") pod "ee7b110a-cf7a-4a6a-91f0-d49900cf3769" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769"). InnerVolumeSpecName "kube-api-access-5xgvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:38:34.047850 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.047822 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ee7b110a-cf7a-4a6a-91f0-d49900cf3769" (UID: "ee7b110a-cf7a-4a6a-91f0-d49900cf3769"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:38:34.140883 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.140837 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-installation-pull-secrets\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:38:34.140883 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.140880 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-image-registry-private-configuration\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:38:34.140883 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.140893 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-tls\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:38:34.140883 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.140901 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-registry-certificates\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:38:34.141157 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.140909 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-bound-sa-token\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:38:34.141157 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.140918 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-ca-trust-extracted\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:38:34.141157 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.140926 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xgvh\" (UniqueName: \"kubernetes.io/projected/ee7b110a-cf7a-4a6a-91f0-d49900cf3769-kube-api-access-5xgvh\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:38:34.746668 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.746623 2563 generic.go:356] "Generic (PLEG): container finished" podID="ee7b110a-cf7a-4a6a-91f0-d49900cf3769" containerID="ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d" exitCode=0 Jul 17 18:38:34.747087 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.746705 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" event={"ID":"ee7b110a-cf7a-4a6a-91f0-d49900cf3769","Type":"ContainerDied","Data":"ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d"} Jul 17 18:38:34.747087 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.746762 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" event={"ID":"ee7b110a-cf7a-4a6a-91f0-d49900cf3769","Type":"ContainerDied","Data":"2f14177aa33c154635f11d8ee7221a0122df87325a0584bfa025a0829058cc4b"} Jul 17 18:38:34.747087 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.746783 2563 scope.go:117] "RemoveContainer" containerID="ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d" Jul 17 18:38:34.747087 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.746716 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67cc6846cb-j9mzl" Jul 17 18:38:34.756269 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.756246 2563 scope.go:117] "RemoveContainer" containerID="ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d" Jul 17 18:38:34.756634 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:38:34.756612 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d\": container with ID starting with ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d not found: ID does not exist" containerID="ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d" Jul 17 18:38:34.756688 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.756645 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d"} err="failed to get container status \"ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d\": rpc error: code = NotFound desc = could not find container \"ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d\": container with ID starting with ff80c0544eec733aedda16fab5618b1ace81a72f7f192aa4e6032a12e2d1065d not found: ID does not exist" Jul 17 18:38:34.768800 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.768769 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67cc6846cb-j9mzl"] Jul 17 18:38:34.774273 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:34.774241 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67cc6846cb-j9mzl"] Jul 17 18:38:35.621384 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:35.621351 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee7b110a-cf7a-4a6a-91f0-d49900cf3769" path="/var/lib/kubelet/pods/ee7b110a-cf7a-4a6a-91f0-d49900cf3769/volumes" Jul 17 18:38:35.752700 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:35.752668 2563 generic.go:356] "Generic (PLEG): container finished" podID="3ff9293e-8e04-4639-b2e8-4e806d121427" containerID="496a3b63a3a4daf633981517749151828b3a2fe1d667566bb1fc42a04ffe04b2" exitCode=0 Jul 17 18:38:35.752700 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:35.752706 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" event={"ID":"3ff9293e-8e04-4639-b2e8-4e806d121427","Type":"ContainerDied","Data":"496a3b63a3a4daf633981517749151828b3a2fe1d667566bb1fc42a04ffe04b2"} Jul 17 18:38:35.753146 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:35.753014 2563 scope.go:117] "RemoveContainer" containerID="496a3b63a3a4daf633981517749151828b3a2fe1d667566bb1fc42a04ffe04b2" Jul 17 18:38:36.758813 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:36.758779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-mt27w" event={"ID":"3ff9293e-8e04-4639-b2e8-4e806d121427","Type":"ContainerStarted","Data":"c603c25bcc4de229f46b0a9627bbbf11eeba2accebad0a9c22ea19714de25c8c"} Jul 17 18:38:47.142316 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:47.142276 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:38:47.146229 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:38:47.146201 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-75c8cfd894-7jmvl" Jul 17 18:39:17.524321 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:17.524260 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:39:17.526802 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:17.526777 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e6936f2-9870-44cc-abd2-99ae76f575ff-metrics-certs\") pod \"network-metrics-daemon-6jqfb\" (UID: \"7e6936f2-9870-44cc-abd2-99ae76f575ff\") " pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:39:17.820735 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:17.820647 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gbnj9\"" Jul 17 18:39:17.828932 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:17.828897 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6jqfb" Jul 17 18:39:17.981698 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:17.981662 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6jqfb"] Jul 17 18:39:17.985479 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:39:17.985423 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6936f2_9870_44cc_abd2_99ae76f575ff.slice/crio-25108f0e601096fbf30d1468eeb5cd0230ec5eab2019752b55d3dd02e02985f3 WatchSource:0}: Error finding container 25108f0e601096fbf30d1468eeb5cd0230ec5eab2019752b55d3dd02e02985f3: Status 404 returned error can't find the container with id 25108f0e601096fbf30d1468eeb5cd0230ec5eab2019752b55d3dd02e02985f3 Jul 17 18:39:18.939004 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:18.938903 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jqfb" event={"ID":"7e6936f2-9870-44cc-abd2-99ae76f575ff","Type":"ContainerStarted","Data":"e747e21cfc1086e87ec103ee7bfc42d9dfa0f2cfa233faf476620bb61e794025"} Jul 17 18:39:18.939004 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:18.938970 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jqfb" event={"ID":"7e6936f2-9870-44cc-abd2-99ae76f575ff","Type":"ContainerStarted","Data":"25108f0e601096fbf30d1468eeb5cd0230ec5eab2019752b55d3dd02e02985f3"} Jul 17 18:39:19.944766 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:19.944731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6jqfb" event={"ID":"7e6936f2-9870-44cc-abd2-99ae76f575ff","Type":"ContainerStarted","Data":"4008e46cb4af45ce084d5d26094c7e17bb99a0e06f0037758e0d38fa9b1add15"} Jul 17 18:39:19.969003 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:19.968944 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6jqfb" podStartSLOduration=375.114963842 podStartE2EDuration="6m15.9688664s" totalImagesPullingTime="853.902558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:39:19.96840411 +0000 UTC m=+376.871977986" watchObservedRunningTime="2026-07-17 18:39:19.9688664 +0000 UTC m=+376.872440279" Jul 17 18:39:39.430007 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.429966 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-689774fb5c-j5668"] Jul 17 18:39:39.430517 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.430272 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee7b110a-cf7a-4a6a-91f0-d49900cf3769" containerName="registry" Jul 17 18:39:39.430517 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.430283 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee7b110a-cf7a-4a6a-91f0-d49900cf3769" containerName="registry" Jul 17 18:39:39.430517 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.430355 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee7b110a-cf7a-4a6a-91f0-d49900cf3769" containerName="registry" Jul 17 18:39:39.433524 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.433502 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.447417 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.447385 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689774fb5c-j5668"] Jul 17 18:39:39.506298 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.506259 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-serving-cert\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.506298 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.506304 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-oauth-config\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.506565 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.506328 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-service-ca\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.506565 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.506418 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh2m7\" (UniqueName: \"kubernetes.io/projected/949f3554-ed03-4dc7-9707-c61348c2c484-kube-api-access-rh2m7\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.506565 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.506489 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-trusted-ca-bundle\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.506565 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.506512 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-console-config\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.506565 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.506526 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-oauth-serving-cert\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.607331 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.607293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2m7\" (UniqueName: \"kubernetes.io/projected/949f3554-ed03-4dc7-9707-c61348c2c484-kube-api-access-rh2m7\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.607519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.607349 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-trusted-ca-bundle\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.607519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.607373 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-console-config\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.607519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.607414 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-oauth-serving-cert\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.607519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.607506 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-serving-cert\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.607714 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.607543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-oauth-config\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.607714 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.607577 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-service-ca\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.608203 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.608170 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-oauth-serving-cert\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.608322 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.608302 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-trusted-ca-bundle\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.608375 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.608307 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-service-ca\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.608433 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.608341 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-console-config\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.610247 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.610223 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-serving-cert\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.610335 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.610249 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-oauth-config\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.617909 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.617882 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2m7\" (UniqueName: \"kubernetes.io/projected/949f3554-ed03-4dc7-9707-c61348c2c484-kube-api-access-rh2m7\") pod \"console-689774fb5c-j5668\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.742424 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.742330 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:39.890074 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:39.890045 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689774fb5c-j5668"] Jul 17 18:39:39.892406 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:39:39.892379 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949f3554_ed03_4dc7_9707_c61348c2c484.slice/crio-8f42decd03818125009abc55d7e3fc4041a4ae7573ad7a7f7cbb60b5982f5cf4 WatchSource:0}: Error finding container 8f42decd03818125009abc55d7e3fc4041a4ae7573ad7a7f7cbb60b5982f5cf4: Status 404 returned error can't find the container with id 8f42decd03818125009abc55d7e3fc4041a4ae7573ad7a7f7cbb60b5982f5cf4 Jul 17 18:39:40.029246 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:40.029154 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689774fb5c-j5668" event={"ID":"949f3554-ed03-4dc7-9707-c61348c2c484","Type":"ContainerStarted","Data":"cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813"} Jul 17 18:39:40.029246 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:40.029205 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689774fb5c-j5668" event={"ID":"949f3554-ed03-4dc7-9707-c61348c2c484","Type":"ContainerStarted","Data":"8f42decd03818125009abc55d7e3fc4041a4ae7573ad7a7f7cbb60b5982f5cf4"} Jul 17 18:39:40.053340 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:40.053286 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-689774fb5c-j5668" podStartSLOduration=1.053269948 podStartE2EDuration="1.053269948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:39:40.051832516 +0000 UTC m=+396.955406394" watchObservedRunningTime="2026-07-17 18:39:40.053269948 +0000 UTC m=+396.956843823" Jul 17 18:39:49.742663 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:49.742609 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:49.743234 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:49.742708 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:49.747725 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:49.747698 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:50.071779 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:50.071693 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:39:50.129726 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:39:50.129691 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bdf799767-jmn7l"] Jul 17 18:40:15.152650 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.152591 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bdf799767-jmn7l" podUID="252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" containerName="console" containerID="cri-o://1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f" gracePeriod=15 Jul 17 18:40:15.410143 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.410118 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:40:15.496593 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.496562 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-oauth-config\") pod \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " Jul 17 18:40:15.496786 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.496617 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-trusted-ca-bundle\") pod \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " Jul 17 18:40:15.496786 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.496642 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq4lx\" (UniqueName: \"kubernetes.io/projected/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-kube-api-access-dq4lx\") pod \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " Jul 17 18:40:15.496786 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.496673 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-service-ca\") pod \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " Jul 17 18:40:15.496786 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.496705 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-serving-cert\") pod \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " Jul 17 18:40:15.496786 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.496744 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-oauth-serving-cert\") pod \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " Jul 17 18:40:15.496786 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.496784 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-config\") pod \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\" (UID: \"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f\") " Jul 17 18:40:15.497169 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.497112 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" (UID: "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:40:15.497298 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.497275 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-service-ca" (OuterVolumeSpecName: "service-ca") pod "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" (UID: "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:40:15.497414 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.497381 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" (UID: "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:40:15.497524 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.497416 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-config" (OuterVolumeSpecName: "console-config") pod "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" (UID: "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:40:15.499145 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.499122 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" (UID: "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:40:15.499500 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.499418 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" (UID: "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:40:15.499500 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.499424 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-kube-api-access-dq4lx" (OuterVolumeSpecName: "kube-api-access-dq4lx") pod "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" (UID: "252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f"). InnerVolumeSpecName "kube-api-access-dq4lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:40:15.598299 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.598264 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-serving-cert\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:40:15.598299 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.598294 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-oauth-serving-cert\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:40:15.598299 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.598303 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-config\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:40:15.598556 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.598313 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-console-oauth-config\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:40:15.598556 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.598321 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-trusted-ca-bundle\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:40:15.598556 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.598329 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq4lx\" (UniqueName: \"kubernetes.io/projected/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-kube-api-access-dq4lx\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:40:15.598556 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:15.598337 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f-service-ca\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:40:16.169614 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:16.169574 2563 generic.go:356] "Generic (PLEG): container finished" podID="252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" containerID="1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f" exitCode=0 Jul 17 18:40:16.169614 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:16.169618 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bdf799767-jmn7l" Jul 17 18:40:16.170046 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:16.169661 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdf799767-jmn7l" event={"ID":"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f","Type":"ContainerDied","Data":"1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f"} Jul 17 18:40:16.170046 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:16.169708 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bdf799767-jmn7l" event={"ID":"252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f","Type":"ContainerDied","Data":"5f820a75eac98c49567487279f2dbb47c8f770b34608e9a32dcfe7c732a45c4c"} Jul 17 18:40:16.170046 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:16.169728 2563 scope.go:117] "RemoveContainer" containerID="1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f" Jul 17 18:40:16.178439 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:16.178413 2563 scope.go:117] "RemoveContainer" containerID="1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f" Jul 17 18:40:16.178784 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:40:16.178765 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f\": container with ID starting with 1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f not found: ID does not exist" containerID="1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f" Jul 17 18:40:16.178838 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:16.178794 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f"} err="failed to get container status \"1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f\": rpc error: code = NotFound desc = could not find container \"1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f\": container with ID starting with 1abd5b00e2dc57427841793270fb6111dab78248624f7cdddcbe78aa2b15712f not found: ID does not exist" Jul 17 18:40:16.190378 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:16.190348 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bdf799767-jmn7l"] Jul 17 18:40:16.194814 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:16.194760 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bdf799767-jmn7l"] Jul 17 18:40:17.622208 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:40:17.622162 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" path="/var/lib/kubelet/pods/252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f/volumes" Jul 17 18:41:15.388304 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.388265 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6wlm9"] Jul 17 18:41:15.388796 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.388593 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" containerName="console" Jul 17 18:41:15.388796 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.388604 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" containerName="console" Jul 17 18:41:15.388796 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.388658 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="252a4ae0-e7df-4ffd-8cbb-69f7c0a1073f" containerName="console" Jul 17 18:41:15.392724 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.392700 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" Jul 17 18:41:15.395334 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.395309 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 18:41:15.395614 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.395595 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 18:41:15.396154 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.396138 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8vwc4\"" Jul 17 18:41:15.402641 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.402615 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6wlm9"] Jul 17 18:41:15.485406 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.485366 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcf0ffe5-293c-49e6-85a8-20e1ced10d43-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6wlm9\" (UID: \"fcf0ffe5-293c-49e6-85a8-20e1ced10d43\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" Jul 17 18:41:15.485605 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.485425 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cjcq\" (UniqueName: \"kubernetes.io/projected/fcf0ffe5-293c-49e6-85a8-20e1ced10d43-kube-api-access-8cjcq\") pod \"cert-manager-webhook-6486ffd997-6wlm9\" (UID: \"fcf0ffe5-293c-49e6-85a8-20e1ced10d43\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" Jul 17 18:41:15.586210 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.586179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cjcq\" (UniqueName: \"kubernetes.io/projected/fcf0ffe5-293c-49e6-85a8-20e1ced10d43-kube-api-access-8cjcq\") pod \"cert-manager-webhook-6486ffd997-6wlm9\" (UID: \"fcf0ffe5-293c-49e6-85a8-20e1ced10d43\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" Jul 17 18:41:15.586344 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.586237 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcf0ffe5-293c-49e6-85a8-20e1ced10d43-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6wlm9\" (UID: \"fcf0ffe5-293c-49e6-85a8-20e1ced10d43\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" Jul 17 18:41:15.598034 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.598002 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fcf0ffe5-293c-49e6-85a8-20e1ced10d43-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6wlm9\" (UID: \"fcf0ffe5-293c-49e6-85a8-20e1ced10d43\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" Jul 17 18:41:15.598208 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.598184 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cjcq\" (UniqueName: \"kubernetes.io/projected/fcf0ffe5-293c-49e6-85a8-20e1ced10d43-kube-api-access-8cjcq\") pod \"cert-manager-webhook-6486ffd997-6wlm9\" (UID: \"fcf0ffe5-293c-49e6-85a8-20e1ced10d43\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" Jul 17 18:41:15.716956 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.716911 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" Jul 17 18:41:15.863989 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:15.863955 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6wlm9"] Jul 17 18:41:15.867247 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:41:15.867217 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcf0ffe5_293c_49e6_85a8_20e1ced10d43.slice/crio-053af0c0100840259de3cf01380c840cf2f2c42a7d3a826d988d032f76abc81a WatchSource:0}: Error finding container 053af0c0100840259de3cf01380c840cf2f2c42a7d3a826d988d032f76abc81a: Status 404 returned error can't find the container with id 053af0c0100840259de3cf01380c840cf2f2c42a7d3a826d988d032f76abc81a Jul 17 18:41:16.405082 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:16.405049 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" event={"ID":"fcf0ffe5-293c-49e6-85a8-20e1ced10d43","Type":"ContainerStarted","Data":"053af0c0100840259de3cf01380c840cf2f2c42a7d3a826d988d032f76abc81a"} Jul 17 18:41:19.421174 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:19.421141 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" event={"ID":"fcf0ffe5-293c-49e6-85a8-20e1ced10d43","Type":"ContainerStarted","Data":"2ae5e04e1774217a7cd885db0075bbbaea8177fc0c3da1a60b207e57e40729af"} Jul 17 18:41:19.421696 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:19.421258 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" Jul 17 18:41:19.442404 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:19.442347 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" podStartSLOduration=1.375416038 podStartE2EDuration="4.44233195s" totalImagesPullingTime="3.066915912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:41:19.440441724 +0000 UTC m=+496.344015602" watchObservedRunningTime="2026-07-17 18:41:19.44233195 +0000 UTC m=+496.345905825" Jul 17 18:41:25.427588 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:25.427551 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6wlm9" Jul 17 18:41:28.160719 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.160680 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm"] Jul 17 18:41:28.163939 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.163921 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" Jul 17 18:41:28.167038 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.167011 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 18:41:28.167215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.167084 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-s7bq7\"" Jul 17 18:41:28.167858 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.167836 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 18:41:28.178614 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.178584 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm"] Jul 17 18:41:28.187555 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.187519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00eed65e-1fe2-441c-839b-075c6ccec781-tmp\") pod \"openshift-lws-operator-fd8ccff4c-85mpm\" (UID: \"00eed65e-1fe2-441c-839b-075c6ccec781\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" Jul 17 18:41:28.187725 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.187594 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pd2t\" (UniqueName: \"kubernetes.io/projected/00eed65e-1fe2-441c-839b-075c6ccec781-kube-api-access-2pd2t\") pod \"openshift-lws-operator-fd8ccff4c-85mpm\" (UID: \"00eed65e-1fe2-441c-839b-075c6ccec781\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" Jul 17 18:41:28.288811 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.288777 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pd2t\" (UniqueName: \"kubernetes.io/projected/00eed65e-1fe2-441c-839b-075c6ccec781-kube-api-access-2pd2t\") pod \"openshift-lws-operator-fd8ccff4c-85mpm\" (UID: \"00eed65e-1fe2-441c-839b-075c6ccec781\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" Jul 17 18:41:28.288999 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.288846 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00eed65e-1fe2-441c-839b-075c6ccec781-tmp\") pod \"openshift-lws-operator-fd8ccff4c-85mpm\" (UID: \"00eed65e-1fe2-441c-839b-075c6ccec781\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" Jul 17 18:41:28.289211 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.289195 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00eed65e-1fe2-441c-839b-075c6ccec781-tmp\") pod \"openshift-lws-operator-fd8ccff4c-85mpm\" (UID: \"00eed65e-1fe2-441c-839b-075c6ccec781\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" Jul 17 18:41:28.297785 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.297751 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pd2t\" (UniqueName: \"kubernetes.io/projected/00eed65e-1fe2-441c-839b-075c6ccec781-kube-api-access-2pd2t\") pod \"openshift-lws-operator-fd8ccff4c-85mpm\" (UID: \"00eed65e-1fe2-441c-839b-075c6ccec781\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" Jul 17 18:41:28.473306 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.473199 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" Jul 17 18:41:28.614600 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:28.614469 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm"] Jul 17 18:41:28.617351 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:41:28.617320 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00eed65e_1fe2_441c_839b_075c6ccec781.slice/crio-d6798746752e46c96c4a10f16c6b4675cef116ca8ce726dc61e5c025280edbc1 WatchSource:0}: Error finding container d6798746752e46c96c4a10f16c6b4675cef116ca8ce726dc61e5c025280edbc1: Status 404 returned error can't find the container with id d6798746752e46c96c4a10f16c6b4675cef116ca8ce726dc61e5c025280edbc1 Jul 17 18:41:29.462431 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:29.462388 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" event={"ID":"00eed65e-1fe2-441c-839b-075c6ccec781","Type":"ContainerStarted","Data":"d6798746752e46c96c4a10f16c6b4675cef116ca8ce726dc61e5c025280edbc1"} Jul 17 18:41:32.477991 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:32.477953 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" event={"ID":"00eed65e-1fe2-441c-839b-075c6ccec781","Type":"ContainerStarted","Data":"22b99e5521eef33d8f513353b3c5c187646cd15a6ed9f90f741bec8b49769c92"} Jul 17 18:41:32.496784 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:32.496726 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-85mpm" podStartSLOduration=1.054008163 podStartE2EDuration="4.496706402s" totalImagesPullingTime="3.442698239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:41:32.494670403 +0000 UTC m=+509.398244278" watchObservedRunningTime="2026-07-17 18:41:32.496706402 +0000 UTC m=+509.400280277" Jul 17 18:41:48.237833 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.237799 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll"] Jul 17 18:41:48.241837 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.241811 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.244418 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.244383 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 18:41:48.244607 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.244551 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 18:41:48.244707 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.244687 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n2nnj\"" Jul 17 18:41:48.244759 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.244691 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 18:41:48.244814 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.244781 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 18:41:48.265984 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.265946 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll"] Jul 17 18:41:48.350018 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.349975 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ghdg\" (UniqueName: \"kubernetes.io/projected/e4467a8e-8f06-4c08-890d-514366edb029-kube-api-access-6ghdg\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.350018 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.350025 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4467a8e-8f06-4c08-890d-514366edb029-tmp\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.350247 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.350051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4467a8e-8f06-4c08-890d-514366edb029-manifests\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.350247 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.350119 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4467a8e-8f06-4c08-890d-514366edb029-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.350247 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.350170 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4467a8e-8f06-4c08-890d-514366edb029-webhook-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.451363 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.451320 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4467a8e-8f06-4c08-890d-514366edb029-webhook-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.451575 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.451387 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ghdg\" (UniqueName: \"kubernetes.io/projected/e4467a8e-8f06-4c08-890d-514366edb029-kube-api-access-6ghdg\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.451575 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.451436 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4467a8e-8f06-4c08-890d-514366edb029-tmp\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.451575 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.451508 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4467a8e-8f06-4c08-890d-514366edb029-manifests\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.451575 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.451539 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4467a8e-8f06-4c08-890d-514366edb029-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.451926 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.451908 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e4467a8e-8f06-4c08-890d-514366edb029-manifests\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.453966 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.453944 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e4467a8e-8f06-4c08-890d-514366edb029-tmp\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.454082 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.454049 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e4467a8e-8f06-4c08-890d-514366edb029-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.454250 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.454226 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4467a8e-8f06-4c08-890d-514366edb029-webhook-cert\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.467186 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.467147 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ghdg\" (UniqueName: \"kubernetes.io/projected/e4467a8e-8f06-4c08-890d-514366edb029-kube-api-access-6ghdg\") pod \"opendatahub-operator-controller-manager-57479f7bfb-qdzll\" (UID: \"e4467a8e-8f06-4c08-890d-514366edb029\") " pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.552846 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.552755 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:48.705169 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:48.705145 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll"] Jul 17 18:41:48.708021 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:41:48.707983 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4467a8e_8f06_4c08_890d_514366edb029.slice/crio-8fb353ebd815ff1c92c742dc7ecbe92a0e9190f495e19ab9600849f05030b787 WatchSource:0}: Error finding container 8fb353ebd815ff1c92c742dc7ecbe92a0e9190f495e19ab9600849f05030b787: Status 404 returned error can't find the container with id 8fb353ebd815ff1c92c742dc7ecbe92a0e9190f495e19ab9600849f05030b787 Jul 17 18:41:49.549796 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:49.549760 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" event={"ID":"e4467a8e-8f06-4c08-890d-514366edb029","Type":"ContainerStarted","Data":"8fb353ebd815ff1c92c742dc7ecbe92a0e9190f495e19ab9600849f05030b787"} Jul 17 18:41:52.566813 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:52.566779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" event={"ID":"e4467a8e-8f06-4c08-890d-514366edb029","Type":"ContainerStarted","Data":"938157b6a93c9f72a4d1fd54f0581f69567f29f49dc4462621456c368556997f"} Jul 17 18:41:53.573191 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:53.573159 2563 generic.go:356] "Generic (PLEG): container finished" podID="e4467a8e-8f06-4c08-890d-514366edb029" containerID="938157b6a93c9f72a4d1fd54f0581f69567f29f49dc4462621456c368556997f" exitCode=0 Jul 17 18:41:53.573191 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:53.573199 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" event={"ID":"e4467a8e-8f06-4c08-890d-514366edb029","Type":"ContainerDied","Data":"938157b6a93c9f72a4d1fd54f0581f69567f29f49dc4462621456c368556997f"} Jul 17 18:41:54.579756 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:54.579718 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" event={"ID":"e4467a8e-8f06-4c08-890d-514366edb029","Type":"ContainerStarted","Data":"86c864c4c38b32da2bdcd5f481ad36b3703cfc20842a8de8e54da9b7baba1cba"} Jul 17 18:41:54.580189 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:54.579885 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:41:54.606511 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:41:54.606432 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" podStartSLOduration=2.153094733 podStartE2EDuration="6.606414238s" totalImagesPullingTime="3.415860651s" totalInitContainerRuntime="1.037458854s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:41:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:41:54.604906123 +0000 UTC m=+531.508479999" watchObservedRunningTime="2026-07-17 18:41:54.606414238 +0000 UTC m=+531.509988113" Jul 17 18:42:05.586837 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:05.586801 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57479f7bfb-qdzll" Jul 17 18:42:08.297359 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.297323 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6prfl"] Jul 17 18:42:08.305164 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.305144 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:08.310897 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.308670 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 17 18:42:08.310897 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.308961 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2z6tj\"" Jul 17 18:42:08.314199 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.314173 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6prfl"] Jul 17 18:42:08.365412 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.365355 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc"] Jul 17 18:42:08.367836 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.367803 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.372082 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.372045 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 17 18:42:08.372082 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.372074 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 17 18:42:08.372257 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.372173 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-85j2c\"" Jul 17 18:42:08.387464 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.387414 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc"] Jul 17 18:42:08.439012 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.438975 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vpf\" (UniqueName: \"kubernetes.io/projected/1aa7760b-8a78-4310-b9ea-3482fcc94ba8-kube-api-access-r9vpf\") pod \"odh-model-controller-7998c67dc4-6prfl\" (UID: \"1aa7760b-8a78-4310-b9ea-3482fcc94ba8\") " pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:08.439012 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.439014 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1aa7760b-8a78-4310-b9ea-3482fcc94ba8-cert\") pod \"odh-model-controller-7998c67dc4-6prfl\" (UID: \"1aa7760b-8a78-4310-b9ea-3482fcc94ba8\") " pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:08.540252 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.540215 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vpf\" (UniqueName: \"kubernetes.io/projected/1aa7760b-8a78-4310-b9ea-3482fcc94ba8-kube-api-access-r9vpf\") pod \"odh-model-controller-7998c67dc4-6prfl\" (UID: \"1aa7760b-8a78-4310-b9ea-3482fcc94ba8\") " pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:08.540252 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.540255 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05328159-9607-4cc5-898b-25388d78633b-tls-certs\") pod \"kube-auth-proxy-864488b8fd-6g5lc\" (UID: \"05328159-9607-4cc5-898b-25388d78633b\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.540573 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.540276 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05328159-9607-4cc5-898b-25388d78633b-tmp\") pod \"kube-auth-proxy-864488b8fd-6g5lc\" (UID: \"05328159-9607-4cc5-898b-25388d78633b\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.540573 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.540294 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1aa7760b-8a78-4310-b9ea-3482fcc94ba8-cert\") pod \"odh-model-controller-7998c67dc4-6prfl\" (UID: \"1aa7760b-8a78-4310-b9ea-3482fcc94ba8\") " pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:08.540573 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.540361 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2q2\" (UniqueName: \"kubernetes.io/projected/05328159-9607-4cc5-898b-25388d78633b-kube-api-access-qw2q2\") pod \"kube-auth-proxy-864488b8fd-6g5lc\" (UID: \"05328159-9607-4cc5-898b-25388d78633b\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.540573 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:42:08.540433 2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 17 18:42:08.540573 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:42:08.540528 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa7760b-8a78-4310-b9ea-3482fcc94ba8-cert podName:1aa7760b-8a78-4310-b9ea-3482fcc94ba8 nodeName:}" failed. No retries permitted until 2026-07-17 18:42:09.040510953 +0000 UTC m=+545.944084807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1aa7760b-8a78-4310-b9ea-3482fcc94ba8-cert") pod "odh-model-controller-7998c67dc4-6prfl" (UID: "1aa7760b-8a78-4310-b9ea-3482fcc94ba8") : secret "odh-model-controller-webhook-cert" not found Jul 17 18:42:08.561236 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.561167 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vpf\" (UniqueName: \"kubernetes.io/projected/1aa7760b-8a78-4310-b9ea-3482fcc94ba8-kube-api-access-r9vpf\") pod \"odh-model-controller-7998c67dc4-6prfl\" (UID: \"1aa7760b-8a78-4310-b9ea-3482fcc94ba8\") " pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:08.641521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.641479 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05328159-9607-4cc5-898b-25388d78633b-tls-certs\") pod \"kube-auth-proxy-864488b8fd-6g5lc\" (UID: \"05328159-9607-4cc5-898b-25388d78633b\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.641521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.641523 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05328159-9607-4cc5-898b-25388d78633b-tmp\") pod \"kube-auth-proxy-864488b8fd-6g5lc\" (UID: \"05328159-9607-4cc5-898b-25388d78633b\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.641784 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.641574 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2q2\" (UniqueName: \"kubernetes.io/projected/05328159-9607-4cc5-898b-25388d78633b-kube-api-access-qw2q2\") pod \"kube-auth-proxy-864488b8fd-6g5lc\" (UID: \"05328159-9607-4cc5-898b-25388d78633b\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.644051 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.644022 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05328159-9607-4cc5-898b-25388d78633b-tmp\") pod \"kube-auth-proxy-864488b8fd-6g5lc\" (UID: \"05328159-9607-4cc5-898b-25388d78633b\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.644278 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.644258 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05328159-9607-4cc5-898b-25388d78633b-tls-certs\") pod \"kube-auth-proxy-864488b8fd-6g5lc\" (UID: \"05328159-9607-4cc5-898b-25388d78633b\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.652766 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.652740 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2q2\" (UniqueName: \"kubernetes.io/projected/05328159-9607-4cc5-898b-25388d78633b-kube-api-access-qw2q2\") pod \"kube-auth-proxy-864488b8fd-6g5lc\" (UID: \"05328159-9607-4cc5-898b-25388d78633b\") " pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.677302 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.677259 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" Jul 17 18:42:08.819434 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:08.819244 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc"] Jul 17 18:42:08.822130 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:42:08.822099 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05328159_9607_4cc5_898b_25388d78633b.slice/crio-85ea61fe0f73c1f4bbf0902f7b97248150e060afd57b1902761b4a5bae6e057c WatchSource:0}: Error finding container 85ea61fe0f73c1f4bbf0902f7b97248150e060afd57b1902761b4a5bae6e057c: Status 404 returned error can't find the container with id 85ea61fe0f73c1f4bbf0902f7b97248150e060afd57b1902761b4a5bae6e057c Jul 17 18:42:09.045425 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.045383 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1aa7760b-8a78-4310-b9ea-3482fcc94ba8-cert\") pod \"odh-model-controller-7998c67dc4-6prfl\" (UID: \"1aa7760b-8a78-4310-b9ea-3482fcc94ba8\") " pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:09.048002 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.047978 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1aa7760b-8a78-4310-b9ea-3482fcc94ba8-cert\") pod \"odh-model-controller-7998c67dc4-6prfl\" (UID: \"1aa7760b-8a78-4310-b9ea-3482fcc94ba8\") " pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:09.175921 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.175888 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zqxbn"] Jul 17 18:42:09.179376 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.179360 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" Jul 17 18:42:09.181594 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.181569 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 17 18:42:09.198779 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.198749 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zqxbn"] Jul 17 18:42:09.222208 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.222176 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:09.347846 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.347814 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4n8v\" (UniqueName: \"kubernetes.io/projected/1d960e12-58ba-4994-b607-2e80aff93084-kube-api-access-b4n8v\") pod \"model-serving-api-54d89bd5fc-zqxbn\" (UID: \"1d960e12-58ba-4994-b607-2e80aff93084\") " pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" Jul 17 18:42:09.348197 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.347865 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d960e12-58ba-4994-b607-2e80aff93084-tls-certs\") pod \"model-serving-api-54d89bd5fc-zqxbn\" (UID: \"1d960e12-58ba-4994-b607-2e80aff93084\") " pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" Jul 17 18:42:09.411736 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.411712 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-6prfl"] Jul 17 18:42:09.414789 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:42:09.414759 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa7760b_8a78_4310_b9ea_3482fcc94ba8.slice/crio-7324f8a48c6caa75cf307f46e7a13657435d5d5b35e9a6da7b19f9c8d144de5c WatchSource:0}: Error finding container 7324f8a48c6caa75cf307f46e7a13657435d5d5b35e9a6da7b19f9c8d144de5c: Status 404 returned error can't find the container with id 7324f8a48c6caa75cf307f46e7a13657435d5d5b35e9a6da7b19f9c8d144de5c Jul 17 18:42:09.448903 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.448815 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4n8v\" (UniqueName: \"kubernetes.io/projected/1d960e12-58ba-4994-b607-2e80aff93084-kube-api-access-b4n8v\") pod \"model-serving-api-54d89bd5fc-zqxbn\" (UID: \"1d960e12-58ba-4994-b607-2e80aff93084\") " pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" Jul 17 18:42:09.448903 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.448872 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d960e12-58ba-4994-b607-2e80aff93084-tls-certs\") pod \"model-serving-api-54d89bd5fc-zqxbn\" (UID: \"1d960e12-58ba-4994-b607-2e80aff93084\") " pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" Jul 17 18:42:09.451397 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.451378 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d960e12-58ba-4994-b607-2e80aff93084-tls-certs\") pod \"model-serving-api-54d89bd5fc-zqxbn\" (UID: \"1d960e12-58ba-4994-b607-2e80aff93084\") " pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" Jul 17 18:42:09.461910 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.461889 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4n8v\" (UniqueName: \"kubernetes.io/projected/1d960e12-58ba-4994-b607-2e80aff93084-kube-api-access-b4n8v\") pod \"model-serving-api-54d89bd5fc-zqxbn\" (UID: \"1d960e12-58ba-4994-b607-2e80aff93084\") " pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" Jul 17 18:42:09.489320 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.489289 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" Jul 17 18:42:09.656298 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.656255 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" event={"ID":"1aa7760b-8a78-4310-b9ea-3482fcc94ba8","Type":"ContainerStarted","Data":"7324f8a48c6caa75cf307f46e7a13657435d5d5b35e9a6da7b19f9c8d144de5c"} Jul 17 18:42:09.657540 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.657505 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" event={"ID":"05328159-9607-4cc5-898b-25388d78633b","Type":"ContainerStarted","Data":"85ea61fe0f73c1f4bbf0902f7b97248150e060afd57b1902761b4a5bae6e057c"} Jul 17 18:42:09.680713 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:09.680634 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zqxbn"] Jul 17 18:42:09.683526 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:42:09.683497 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d960e12_58ba_4994_b607_2e80aff93084.slice/crio-2eeeaee264651a983ac6765c7bea7098241bf53735a4c5ba32245773c93dcf08 WatchSource:0}: Error finding container 2eeeaee264651a983ac6765c7bea7098241bf53735a4c5ba32245773c93dcf08: Status 404 returned error can't find the container with id 2eeeaee264651a983ac6765c7bea7098241bf53735a4c5ba32245773c93dcf08 Jul 17 18:42:10.665462 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:10.665404 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" event={"ID":"1d960e12-58ba-4994-b607-2e80aff93084","Type":"ContainerStarted","Data":"2eeeaee264651a983ac6765c7bea7098241bf53735a4c5ba32245773c93dcf08"} Jul 17 18:42:10.892081 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:10.890509 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5"] Jul 17 18:42:10.893386 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:10.893364 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:10.908249 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:10.906059 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 17 18:42:10.908249 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:10.906326 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 17 18:42:10.908249 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:10.906588 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-d5ww9\"" Jul 17 18:42:10.908249 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:10.906830 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 18:42:10.908249 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:10.907054 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 17 18:42:10.928942 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:10.928909 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5"] Jul 17 18:42:11.062718 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.062676 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-cacerts\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.062899 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.062738 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9trt\" (UniqueName: \"kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-kube-api-access-v9trt\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.062899 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.062774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.062899 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.062809 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-local-certs\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.062899 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.062841 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-token\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.063114 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.062935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.063114 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.062962 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.164147 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.164111 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-cacerts\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.164147 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.164153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9trt\" (UniqueName: \"kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-kube-api-access-v9trt\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.164382 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.164174 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.164382 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.164197 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-local-certs\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.164382 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.164227 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-token\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.164382 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.164302 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.164382 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.164327 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.165215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.165160 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.167570 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.167427 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.167680 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.167580 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-local-certs\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.167795 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.167707 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-cacerts\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.168005 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.167984 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.174903 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.174826 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9trt\" (UniqueName: \"kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-kube-api-access-v9trt\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.175029 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.174938 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-token\") pod \"istiod-openshift-gateway-7574459656-7bgz5\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:11.226682 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:11.226650 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:13.472637 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:13.472601 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5"] Jul 17 18:42:13.473810 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:42:13.473747 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323ffc8c_a64c_4f7c_bea7_b8dcfa4bc464.slice/crio-38032ab8c1943729e6568293e5cbba722962360bceed4988d59c3b68ec96a810 WatchSource:0}: Error finding container 38032ab8c1943729e6568293e5cbba722962360bceed4988d59c3b68ec96a810: Status 404 returned error can't find the container with id 38032ab8c1943729e6568293e5cbba722962360bceed4988d59c3b68ec96a810 Jul 17 18:42:13.683659 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:13.683625 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" event={"ID":"1d960e12-58ba-4994-b607-2e80aff93084","Type":"ContainerStarted","Data":"1e085e06d0e0c3d973890971f8b4e4eadef48fba1980e48bd0f089bfac62810e"} Jul 17 18:42:13.683874 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:13.683716 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" Jul 17 18:42:13.685419 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:13.685385 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" event={"ID":"1aa7760b-8a78-4310-b9ea-3482fcc94ba8","Type":"ContainerStarted","Data":"b79bd0a63961d6a29ee29e107db417eeca6de2a253844b7580f05666ae49a7ff"} Jul 17 18:42:13.685584 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:13.685475 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:13.686986 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:13.686956 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" event={"ID":"05328159-9607-4cc5-898b-25388d78633b","Type":"ContainerStarted","Data":"c3c22465efbfd2ca9028af29223e3e0a933a0dbf328cc4d518d00918f07781e0"} Jul 17 18:42:13.688307 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:13.688285 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" event={"ID":"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464","Type":"ContainerStarted","Data":"38032ab8c1943729e6568293e5cbba722962360bceed4988d59c3b68ec96a810"} Jul 17 18:42:13.706895 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:13.706846 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" podStartSLOduration=1.079093071 podStartE2EDuration="4.706830254s" totalImagesPullingTime="3.627737183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:13.706138293 +0000 UTC m=+550.609712168" watchObservedRunningTime="2026-07-17 18:42:13.706830254 +0000 UTC m=+550.610404130" Jul 17 18:42:13.726143 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:13.726093 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" podStartSLOduration=1.8238741649999999 podStartE2EDuration="5.726078285s" totalImagesPullingTime="3.90220412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:13.724463074 +0000 UTC m=+550.628036940" watchObservedRunningTime="2026-07-17 18:42:13.726078285 +0000 UTC m=+550.629652161" Jul 17 18:42:13.743545 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:13.743486 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-864488b8fd-6g5lc" podStartSLOduration=1.247219405 podStartE2EDuration="5.743468759s" totalImagesPullingTime="4.496249354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:13.741892083 +0000 UTC m=+550.645465959" watchObservedRunningTime="2026-07-17 18:42:13.743468759 +0000 UTC m=+550.647042638" Jul 17 18:42:16.080567 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:16.080531 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:42:16.080831 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:16.080605 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:42:16.707401 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:16.707359 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" event={"ID":"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464","Type":"ContainerStarted","Data":"838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7"} Jul 17 18:42:16.707616 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:16.707483 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:17.569057 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.569001 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" podStartSLOduration=4.964596695 podStartE2EDuration="7.568980036s" totalImagesPullingTime="2.604383341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:16.752178182 +0000 UTC m=+553.655752103" watchObservedRunningTime="2026-07-17 18:42:17.568980036 +0000 UTC m=+554.472553911" Jul 17 18:42:17.569782 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.569755 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9"] Jul 17 18:42:17.572548 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.572527 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.578504 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.578468 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gs656\"" Jul 17 18:42:17.592757 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.592727 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9"] Jul 17 18:42:17.718170 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.718139 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:17.724830 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.724802 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.725006 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.724835 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.725006 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.724878 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnk7\" (UniqueName: \"kubernetes.io/projected/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-kube-api-access-qmnk7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.725006 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.724987 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.725161 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.725027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.725324 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.725303 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.725495 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.725363 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.725495 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.725399 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.725495 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.725429 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.826657 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.826573 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.826657 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.826613 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.826657 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.826629 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.826657 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.826644 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.826960 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.826680 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.826960 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.826705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.826960 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.826770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnk7\" (UniqueName: \"kubernetes.io/projected/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-kube-api-access-qmnk7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.826960 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.826812 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.826960 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.826845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.827209 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.827030 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.827209 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.827112 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.827411 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.827369 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.827585 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.827563 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.827975 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.827950 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.829253 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.829236 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.829665 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.829647 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.838236 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.838214 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.850144 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.850115 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnk7\" (UniqueName: \"kubernetes.io/projected/5b252de0-39c6-4fd8-9bf6-a9cee59a6c33-kube-api-access-qmnk7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf484jg9\" (UID: \"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:17.885700 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:17.885656 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:18.069805 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:18.069647 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9"] Jul 17 18:42:18.071951 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:42:18.071924 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b252de0_39c6_4fd8_9bf6_a9cee59a6c33.slice/crio-d85026d56d3ec541a8673080dd551dc035fcfdc78427e9ae457a8e534d6b3bdd WatchSource:0}: Error finding container d85026d56d3ec541a8673080dd551dc035fcfdc78427e9ae457a8e534d6b3bdd: Status 404 returned error can't find the container with id d85026d56d3ec541a8673080dd551dc035fcfdc78427e9ae457a8e534d6b3bdd Jul 17 18:42:18.720821 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:18.720785 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" event={"ID":"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33","Type":"ContainerStarted","Data":"d85026d56d3ec541a8673080dd551dc035fcfdc78427e9ae457a8e534d6b3bdd"} Jul 17 18:42:20.069705 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.069668 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs"] Jul 17 18:42:20.072840 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.072819 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:42:20.076670 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.076647 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 17 18:42:20.077035 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.077015 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-f7gss\"" Jul 17 18:42:20.086150 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.086125 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm"] Jul 17 18:42:20.088896 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.088881 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" Jul 17 18:42:20.091815 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.091781 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 17 18:42:20.092203 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.092041 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-snpgs\"" Jul 17 18:42:20.094309 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.094287 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs"] Jul 17 18:42:20.127756 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.127729 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm"] Jul 17 18:42:20.150059 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.150020 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7hjk\" (UniqueName: \"kubernetes.io/projected/0db86b23-a5c6-4919-b36b-d05d33d44e6c-kube-api-access-g7hjk\") pod \"kserve-controller-manager-d6f4b65b8-5c8qs\" (UID: \"0db86b23-a5c6-4919-b36b-d05d33d44e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:42:20.150219 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.150070 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0db86b23-a5c6-4919-b36b-d05d33d44e6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-5c8qs\" (UID: \"0db86b23-a5c6-4919-b36b-d05d33d44e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:42:20.251362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.251324 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwqj8\" (UniqueName: \"kubernetes.io/projected/7a768798-edd2-49f7-81e6-8eb5d880c6f2-kube-api-access-kwqj8\") pod \"llmisvc-controller-manager-777c6bd7d5-7bqrm\" (UID: \"7a768798-edd2-49f7-81e6-8eb5d880c6f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" Jul 17 18:42:20.251362 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.251370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a768798-edd2-49f7-81e6-8eb5d880c6f2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7bqrm\" (UID: \"7a768798-edd2-49f7-81e6-8eb5d880c6f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" Jul 17 18:42:20.251607 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.251440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7hjk\" (UniqueName: \"kubernetes.io/projected/0db86b23-a5c6-4919-b36b-d05d33d44e6c-kube-api-access-g7hjk\") pod \"kserve-controller-manager-d6f4b65b8-5c8qs\" (UID: \"0db86b23-a5c6-4919-b36b-d05d33d44e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:42:20.251607 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.251502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0db86b23-a5c6-4919-b36b-d05d33d44e6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-5c8qs\" (UID: \"0db86b23-a5c6-4919-b36b-d05d33d44e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:42:20.251675 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:42:20.251617 2563 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 17 18:42:20.251675 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:42:20.251671 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0db86b23-a5c6-4919-b36b-d05d33d44e6c-cert podName:0db86b23-a5c6-4919-b36b-d05d33d44e6c nodeName:}" failed. No retries permitted until 2026-07-17 18:42:20.751651719 +0000 UTC m=+557.655225573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0db86b23-a5c6-4919-b36b-d05d33d44e6c-cert") pod "kserve-controller-manager-d6f4b65b8-5c8qs" (UID: "0db86b23-a5c6-4919-b36b-d05d33d44e6c") : secret "kserve-webhook-server-cert" not found Jul 17 18:42:20.279339 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.279304 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7hjk\" (UniqueName: \"kubernetes.io/projected/0db86b23-a5c6-4919-b36b-d05d33d44e6c-kube-api-access-g7hjk\") pod \"kserve-controller-manager-d6f4b65b8-5c8qs\" (UID: \"0db86b23-a5c6-4919-b36b-d05d33d44e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:42:20.352640 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.352549 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwqj8\" (UniqueName: \"kubernetes.io/projected/7a768798-edd2-49f7-81e6-8eb5d880c6f2-kube-api-access-kwqj8\") pod \"llmisvc-controller-manager-777c6bd7d5-7bqrm\" (UID: \"7a768798-edd2-49f7-81e6-8eb5d880c6f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" Jul 17 18:42:20.352640 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.352591 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a768798-edd2-49f7-81e6-8eb5d880c6f2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7bqrm\" (UID: \"7a768798-edd2-49f7-81e6-8eb5d880c6f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" Jul 17 18:42:20.355154 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.355126 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a768798-edd2-49f7-81e6-8eb5d880c6f2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7bqrm\" (UID: \"7a768798-edd2-49f7-81e6-8eb5d880c6f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" Jul 17 18:42:20.407955 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.407925 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwqj8\" (UniqueName: \"kubernetes.io/projected/7a768798-edd2-49f7-81e6-8eb5d880c6f2-kube-api-access-kwqj8\") pod \"llmisvc-controller-manager-777c6bd7d5-7bqrm\" (UID: \"7a768798-edd2-49f7-81e6-8eb5d880c6f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" Jul 17 18:42:20.699642 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.699608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" Jul 17 18:42:20.755016 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.754980 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0db86b23-a5c6-4919-b36b-d05d33d44e6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-5c8qs\" (UID: \"0db86b23-a5c6-4919-b36b-d05d33d44e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:42:20.759689 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.759663 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0db86b23-a5c6-4919-b36b-d05d33d44e6c-cert\") pod \"kserve-controller-manager-d6f4b65b8-5c8qs\" (UID: \"0db86b23-a5c6-4919-b36b-d05d33d44e6c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:42:20.866904 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.866877 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm"] Jul 17 18:42:20.868919 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:42:20.868889 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7a768798_edd2_49f7_81e6_8eb5d880c6f2.slice/crio-9f74e59a90f9e597f926c169281820791c16cac1b9c8098aca76057876f77e3c WatchSource:0}: Error finding container 9f74e59a90f9e597f926c169281820791c16cac1b9c8098aca76057876f77e3c: Status 404 returned error can't find the container with id 9f74e59a90f9e597f926c169281820791c16cac1b9c8098aca76057876f77e3c Jul 17 18:42:20.983188 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:20.983101 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:42:21.138811 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:21.138783 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs"] Jul 17 18:42:21.141969 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:42:21.141934 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db86b23_a5c6_4919_b36b_d05d33d44e6c.slice/crio-8b565fab62fabf4be6f3aa4f7ce46a8fd9da8dfb71240ea482e9293c42ca72ae WatchSource:0}: Error finding container 8b565fab62fabf4be6f3aa4f7ce46a8fd9da8dfb71240ea482e9293c42ca72ae: Status 404 returned error can't find the container with id 8b565fab62fabf4be6f3aa4f7ce46a8fd9da8dfb71240ea482e9293c42ca72ae Jul 17 18:42:21.667422 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:21.666350 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5"] Jul 17 18:42:21.667422 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:21.666661 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" podUID="323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" containerName="discovery" containerID="cri-o://838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7" gracePeriod=30 Jul 17 18:42:21.737658 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:21.737616 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" event={"ID":"7a768798-edd2-49f7-81e6-8eb5d880c6f2","Type":"ContainerStarted","Data":"9f74e59a90f9e597f926c169281820791c16cac1b9c8098aca76057876f77e3c"} Jul 17 18:42:21.739149 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:21.739115 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" event={"ID":"0db86b23-a5c6-4919-b36b-d05d33d44e6c","Type":"ContainerStarted","Data":"8b565fab62fabf4be6f3aa4f7ce46a8fd9da8dfb71240ea482e9293c42ca72ae"} Jul 17 18:42:21.972851 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:21.972813 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:22.070005 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.069968 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9trt\" (UniqueName: \"kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-kube-api-access-v9trt\") pod \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " Jul 17 18:42:22.070199 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.070043 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-local-certs\") pod \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " Jul 17 18:42:22.070199 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.070074 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-ca-configmap\") pod \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " Jul 17 18:42:22.070199 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.070110 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-kubeconfig\") pod \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " Jul 17 18:42:22.070199 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.070184 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-cacerts\") pod \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " Jul 17 18:42:22.070419 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.070235 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-token\") pod \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " Jul 17 18:42:22.070419 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.070264 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-dns-cert\") pod \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\" (UID: \"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464\") " Jul 17 18:42:22.070892 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.070842 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" (UID: "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:42:22.073519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.073468 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" (UID: "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:42:22.073519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.073508 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-local-certs" (OuterVolumeSpecName: "local-certs") pod "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" (UID: "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:42:22.073928 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.073891 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-cacerts" (OuterVolumeSpecName: "cacerts") pod "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" (UID: "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:42:22.074373 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.074348 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" (UID: "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:42:22.074588 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.074567 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-token" (OuterVolumeSpecName: "istio-token") pod "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" (UID: "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:42:22.075139 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.075120 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-kube-api-access-v9trt" (OuterVolumeSpecName: "kube-api-access-v9trt") pod "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" (UID: "323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464"). InnerVolumeSpecName "kube-api-access-v9trt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:42:22.171182 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.171142 2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-kubeconfig\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:42:22.171182 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.171174 2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-cacerts\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:42:22.171182 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.171184 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-token\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:42:22.171182 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.171193 2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-dns-cert\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:42:22.171862 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.171202 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9trt\" (UniqueName: \"kubernetes.io/projected/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-kube-api-access-v9trt\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:42:22.171862 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.171210 2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-local-certs\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:42:22.171862 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.171218 2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464-istio-csr-ca-configmap\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:42:22.747380 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.747340 2563 generic.go:356] "Generic (PLEG): container finished" podID="323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" containerID="838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7" exitCode=0 Jul 17 18:42:22.747597 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.747403 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" Jul 17 18:42:22.747597 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.747466 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" event={"ID":"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464","Type":"ContainerDied","Data":"838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7"} Jul 17 18:42:22.747597 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.747526 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5" event={"ID":"323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464","Type":"ContainerDied","Data":"38032ab8c1943729e6568293e5cbba722962360bceed4988d59c3b68ec96a810"} Jul 17 18:42:22.747597 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.747556 2563 scope.go:117] "RemoveContainer" containerID="838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7" Jul 17 18:42:22.759142 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.759118 2563 scope.go:117] "RemoveContainer" containerID="838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7" Jul 17 18:42:22.759518 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:42:22.759497 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7\": container with ID starting with 838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7 not found: ID does not exist" containerID="838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7" Jul 17 18:42:22.759593 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.759529 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7"} err="failed to get container status \"838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7\": rpc error: code = NotFound desc = could not find container \"838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7\": container with ID starting with 838c5a58379d76f53aa2b048cd9755c4eaf7ab8eaf014cf0e952dccfd99bcbf7 not found: ID does not exist" Jul 17 18:42:22.795470 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.795405 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5"] Jul 17 18:42:22.808612 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:22.808435 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7bgz5"] Jul 17 18:42:23.625104 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:23.625072 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" path="/var/lib/kubelet/pods/323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464/volumes" Jul 17 18:42:24.696728 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:24.696635 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-6prfl" Jul 17 18:42:24.698797 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:24.698771 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zqxbn" Jul 17 18:42:24.759872 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:24.759833 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" event={"ID":"7a768798-edd2-49f7-81e6-8eb5d880c6f2","Type":"ContainerStarted","Data":"beef33a6c37ab6acff1ad54ea1802f922b3d8e9e70b00be3b4d0b3f40adfd4fd"} Jul 17 18:42:24.760063 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:24.759929 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" Jul 17 18:42:24.761654 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:24.761622 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" event={"ID":"0db86b23-a5c6-4919-b36b-d05d33d44e6c","Type":"ContainerStarted","Data":"efe6d3c2eac549233f36180f39161f875f6267e6bc5c50b1859c5bb1e2a9c7ed"} Jul 17 18:42:24.761812 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:24.761687 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:42:24.824921 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:24.824866 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" podStartSLOduration=1.535438157 podStartE2EDuration="4.824849128s" totalImagesPullingTime="3.289410971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:24.823997522 +0000 UTC m=+561.727571397" watchObservedRunningTime="2026-07-17 18:42:24.824849128 +0000 UTC m=+561.728422981" Jul 17 18:42:24.865929 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:24.865861 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" podStartSLOduration=1.299497469 podStartE2EDuration="4.865837212s" totalImagesPullingTime="3.566339743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:24.864111158 +0000 UTC m=+561.767685033" watchObservedRunningTime="2026-07-17 18:42:24.865837212 +0000 UTC m=+561.769411089" Jul 17 18:42:29.673132 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:29.673097 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:42:29.673677 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:29.673186 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:42:29.673677 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:29.673218 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:42:29.791104 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:29.791069 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" event={"ID":"5b252de0-39c6-4fd8-9bf6-a9cee59a6c33","Type":"ContainerStarted","Data":"65114f3ade2f00fcbf0316358f7e21717c7e235656f2ff7771ef264bf7182ed7"} Jul 17 18:42:29.815339 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:29.815284 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" podStartSLOduration=1.216374514 podStartE2EDuration="12.815266922s" totalImagesPullingTime="11.598892408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:42:29.814307982 +0000 UTC m=+566.717881871" watchObservedRunningTime="2026-07-17 18:42:29.815266922 +0000 UTC m=+566.718840798" Jul 17 18:42:29.885989 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:29.885947 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:30.890845 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:30.890813 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:31.802190 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:31.802155 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:31.803156 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:31.803136 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf484jg9" Jul 17 18:42:55.769398 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:55.769362 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7bqrm" Jul 17 18:42:55.772238 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:42:55.772206 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5c8qs" Jul 17 18:43:08.885111 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.885072 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wxckj"] Jul 17 18:43:08.886602 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.885495 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" containerName="discovery" Jul 17 18:43:08.886602 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.885508 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" containerName="discovery" Jul 17 18:43:08.886602 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.885569 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="323ffc8c-a64c-4f7c-bea7-b8dcfa4bc464" containerName="discovery" Jul 17 18:43:08.887519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.887495 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wxckj" Jul 17 18:43:08.890172 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.890148 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 18:43:08.890950 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.890919 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 18:43:08.891064 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.890954 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-r9nzb\"" Jul 17 18:43:08.895910 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.895883 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8822\" (UniqueName: \"kubernetes.io/projected/46b95b17-b9fa-40d6-be11-33d99cc69488-kube-api-access-x8822\") pod \"kuadrant-operator-catalog-wxckj\" (UID: \"46b95b17-b9fa-40d6-be11-33d99cc69488\") " pod="kuadrant-system/kuadrant-operator-catalog-wxckj" Jul 17 18:43:08.904588 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.904553 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wxckj"] Jul 17 18:43:08.997224 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:08.997178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8822\" (UniqueName: \"kubernetes.io/projected/46b95b17-b9fa-40d6-be11-33d99cc69488-kube-api-access-x8822\") pod \"kuadrant-operator-catalog-wxckj\" (UID: \"46b95b17-b9fa-40d6-be11-33d99cc69488\") " pod="kuadrant-system/kuadrant-operator-catalog-wxckj" Jul 17 18:43:09.006605 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.006570 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8822\" (UniqueName: \"kubernetes.io/projected/46b95b17-b9fa-40d6-be11-33d99cc69488-kube-api-access-x8822\") pod \"kuadrant-operator-catalog-wxckj\" (UID: \"46b95b17-b9fa-40d6-be11-33d99cc69488\") " pod="kuadrant-system/kuadrant-operator-catalog-wxckj" Jul 17 18:43:09.198618 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.198576 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wxckj" Jul 17 18:43:09.235082 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.235041 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wxckj"] Jul 17 18:43:09.347277 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.347250 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wxckj"] Jul 17 18:43:09.348956 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:43:09.348930 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b95b17_b9fa_40d6_be11_33d99cc69488.slice/crio-a88495ee13f3d7424fb53dfad530a44031b0bf80f1ce28ba4a5cadf0a70e2ef0 WatchSource:0}: Error finding container a88495ee13f3d7424fb53dfad530a44031b0bf80f1ce28ba4a5cadf0a70e2ef0: Status 404 returned error can't find the container with id a88495ee13f3d7424fb53dfad530a44031b0bf80f1ce28ba4a5cadf0a70e2ef0 Jul 17 18:43:09.350496 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.350476 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:43:09.446212 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.446178 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hzcmh"] Jul 17 18:43:09.449431 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.449375 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" Jul 17 18:43:09.460277 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.460243 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hzcmh"] Jul 17 18:43:09.502113 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.502074 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np4qp\" (UniqueName: \"kubernetes.io/projected/f0b5eef6-8b4f-40df-9188-2bd00ca24189-kube-api-access-np4qp\") pod \"kuadrant-operator-catalog-hzcmh\" (UID: \"f0b5eef6-8b4f-40df-9188-2bd00ca24189\") " pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" Jul 17 18:43:09.602733 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.602687 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np4qp\" (UniqueName: \"kubernetes.io/projected/f0b5eef6-8b4f-40df-9188-2bd00ca24189-kube-api-access-np4qp\") pod \"kuadrant-operator-catalog-hzcmh\" (UID: \"f0b5eef6-8b4f-40df-9188-2bd00ca24189\") " pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" Jul 17 18:43:09.612462 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.612422 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np4qp\" (UniqueName: \"kubernetes.io/projected/f0b5eef6-8b4f-40df-9188-2bd00ca24189-kube-api-access-np4qp\") pod \"kuadrant-operator-catalog-hzcmh\" (UID: \"f0b5eef6-8b4f-40df-9188-2bd00ca24189\") " pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" Jul 17 18:43:09.759617 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.759533 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" Jul 17 18:43:09.919020 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.918992 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hzcmh"] Jul 17 18:43:09.920545 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:43:09.920517 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b5eef6_8b4f_40df_9188_2bd00ca24189.slice/crio-3d9f4b33515d180d073a0fcdb8708a7cd582d1427870e66a2b6e906bf336e1ed WatchSource:0}: Error finding container 3d9f4b33515d180d073a0fcdb8708a7cd582d1427870e66a2b6e906bf336e1ed: Status 404 returned error can't find the container with id 3d9f4b33515d180d073a0fcdb8708a7cd582d1427870e66a2b6e906bf336e1ed Jul 17 18:43:09.985969 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.985936 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" event={"ID":"f0b5eef6-8b4f-40df-9188-2bd00ca24189","Type":"ContainerStarted","Data":"3d9f4b33515d180d073a0fcdb8708a7cd582d1427870e66a2b6e906bf336e1ed"} Jul 17 18:43:09.987140 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:09.987118 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wxckj" event={"ID":"46b95b17-b9fa-40d6-be11-33d99cc69488","Type":"ContainerStarted","Data":"a88495ee13f3d7424fb53dfad530a44031b0bf80f1ce28ba4a5cadf0a70e2ef0"} Jul 17 18:43:12.001576 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:12.001526 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" event={"ID":"f0b5eef6-8b4f-40df-9188-2bd00ca24189","Type":"ContainerStarted","Data":"eaab7fe341ecb72cd994c9b95e1daaad3383110c128d1ad33379c431fb1f770f"} Jul 17 18:43:12.003427 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:12.003388 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wxckj" event={"ID":"46b95b17-b9fa-40d6-be11-33d99cc69488","Type":"ContainerStarted","Data":"bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386"} Jul 17 18:43:12.003627 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:12.003576 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-wxckj" podUID="46b95b17-b9fa-40d6-be11-33d99cc69488" containerName="registry-server" containerID="cri-o://bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386" gracePeriod=2 Jul 17 18:43:12.024746 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:12.024694 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" podStartSLOduration=1.030088186 podStartE2EDuration="3.024677544s" totalImagesPullingTime="1.994589358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:43:12.023686567 +0000 UTC m=+608.927260444" watchObservedRunningTime="2026-07-17 18:43:12.024677544 +0000 UTC m=+608.928251419" Jul 17 18:43:12.052311 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:12.052254 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wxckj" podStartSLOduration=1.490034378 podStartE2EDuration="4.052233896s" totalImagesPullingTime="2.562199518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:43:12.050711473 +0000 UTC m=+608.954285350" watchObservedRunningTime="2026-07-17 18:43:12.052233896 +0000 UTC m=+608.955807771" Jul 17 18:43:12.266596 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:12.266569 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wxckj_46b95b17-b9fa-40d6-be11-33d99cc69488/registry-server/0.log" Jul 17 18:43:12.268930 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:12.268911 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wxckj" Jul 17 18:43:12.329624 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:12.329582 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8822\" (UniqueName: \"kubernetes.io/projected/46b95b17-b9fa-40d6-be11-33d99cc69488-kube-api-access-x8822\") pod \"46b95b17-b9fa-40d6-be11-33d99cc69488\" (UID: \"46b95b17-b9fa-40d6-be11-33d99cc69488\") " Jul 17 18:43:12.331986 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:12.331955 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b95b17-b9fa-40d6-be11-33d99cc69488-kube-api-access-x8822" (OuterVolumeSpecName: "kube-api-access-x8822") pod "46b95b17-b9fa-40d6-be11-33d99cc69488" (UID: "46b95b17-b9fa-40d6-be11-33d99cc69488"). InnerVolumeSpecName "kube-api-access-x8822". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:43:12.431146 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:12.431050 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8822\" (UniqueName: \"kubernetes.io/projected/46b95b17-b9fa-40d6-be11-33d99cc69488-kube-api-access-x8822\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:43:13.009249 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.009221 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wxckj_46b95b17-b9fa-40d6-be11-33d99cc69488/registry-server/0.log" Jul 17 18:43:13.009709 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.009265 2563 generic.go:356] "Generic (PLEG): container finished" podID="46b95b17-b9fa-40d6-be11-33d99cc69488" containerID="bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386" exitCode=2 Jul 17 18:43:13.009709 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.009323 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wxckj" Jul 17 18:43:13.009709 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.009365 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wxckj" event={"ID":"46b95b17-b9fa-40d6-be11-33d99cc69488","Type":"ContainerDied","Data":"bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386"} Jul 17 18:43:13.009709 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.009415 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wxckj" event={"ID":"46b95b17-b9fa-40d6-be11-33d99cc69488","Type":"ContainerDied","Data":"a88495ee13f3d7424fb53dfad530a44031b0bf80f1ce28ba4a5cadf0a70e2ef0"} Jul 17 18:43:13.009709 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.009437 2563 scope.go:117] "RemoveContainer" containerID="bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386" Jul 17 18:43:13.018995 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.018974 2563 scope.go:117] "RemoveContainer" containerID="bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386" Jul 17 18:43:13.019315 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:43:13.019296 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386\": container with ID starting with bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386 not found: ID does not exist" containerID="bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386" Jul 17 18:43:13.019363 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.019328 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386"} err="failed to get container status \"bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386\": rpc error: code = NotFound desc = could not find container \"bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386\": container with ID starting with bb87dd8c454a46c01a9855cf8d482be89b1ff9f79ef9a133a511b1945ce5b386 not found: ID does not exist" Jul 17 18:43:13.030958 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.030920 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wxckj"] Jul 17 18:43:13.034744 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.034715 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wxckj"] Jul 17 18:43:13.622383 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:13.622332 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b95b17-b9fa-40d6-be11-33d99cc69488" path="/var/lib/kubelet/pods/46b95b17-b9fa-40d6-be11-33d99cc69488/volumes" Jul 17 18:43:19.760223 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:19.760185 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" Jul 17 18:43:19.760718 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:19.760274 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" Jul 17 18:43:19.784646 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:19.784618 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" Jul 17 18:43:20.068301 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:20.068223 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-hzcmh" Jul 17 18:43:34.776414 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:34.776328 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98"] Jul 17 18:43:34.776915 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:34.776798 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b95b17-b9fa-40d6-be11-33d99cc69488" containerName="registry-server" Jul 17 18:43:34.776915 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:34.776813 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b95b17-b9fa-40d6-be11-33d99cc69488" containerName="registry-server" Jul 17 18:43:34.776915 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:34.776894 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b95b17-b9fa-40d6-be11-33d99cc69488" containerName="registry-server" Jul 17 18:43:34.780079 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:34.780060 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98" Jul 17 18:43:34.790653 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:34.790626 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-dn5hq\"" Jul 17 18:43:34.791092 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:34.791079 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 17 18:43:34.808052 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:34.808021 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98"] Jul 17 18:43:34.932906 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:34.932863 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztg46\" (UniqueName: \"kubernetes.io/projected/821d5ca9-2299-4fdc-92d4-635f95617da3-kube-api-access-ztg46\") pod \"dns-operator-controller-manager-864dd78c4c-hlb98\" (UID: \"821d5ca9-2299-4fdc-92d4-635f95617da3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98" Jul 17 18:43:35.034479 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:35.034346 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztg46\" (UniqueName: \"kubernetes.io/projected/821d5ca9-2299-4fdc-92d4-635f95617da3-kube-api-access-ztg46\") pod \"dns-operator-controller-manager-864dd78c4c-hlb98\" (UID: \"821d5ca9-2299-4fdc-92d4-635f95617da3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98" Jul 17 18:43:35.045176 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:35.045145 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztg46\" (UniqueName: \"kubernetes.io/projected/821d5ca9-2299-4fdc-92d4-635f95617da3-kube-api-access-ztg46\") pod \"dns-operator-controller-manager-864dd78c4c-hlb98\" (UID: \"821d5ca9-2299-4fdc-92d4-635f95617da3\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98" Jul 17 18:43:35.090841 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:35.090807 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98" Jul 17 18:43:35.295561 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:43:35.295525 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821d5ca9_2299_4fdc_92d4_635f95617da3.slice/crio-81e76f4894af6e9d110f730f4cbd810ca966acdd9a29f9dc5007e0e5dacfef56 WatchSource:0}: Error finding container 81e76f4894af6e9d110f730f4cbd810ca966acdd9a29f9dc5007e0e5dacfef56: Status 404 returned error can't find the container with id 81e76f4894af6e9d110f730f4cbd810ca966acdd9a29f9dc5007e0e5dacfef56 Jul 17 18:43:35.296213 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:35.296168 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98"] Jul 17 18:43:36.121837 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:36.121794 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98" event={"ID":"821d5ca9-2299-4fdc-92d4-635f95617da3","Type":"ContainerStarted","Data":"81e76f4894af6e9d110f730f4cbd810ca966acdd9a29f9dc5007e0e5dacfef56"} Jul 17 18:43:38.137322 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.137235 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98" event={"ID":"821d5ca9-2299-4fdc-92d4-635f95617da3","Type":"ContainerStarted","Data":"f5a9bb571509e268dbd9a9986b15a56c5139f29504d6f8d10042e9fe6762a9fa"} Jul 17 18:43:38.137805 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.137363 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98" Jul 17 18:43:38.155730 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.155680 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98" podStartSLOduration=1.777531725 podStartE2EDuration="4.155664054s" totalImagesPullingTime="2.378132329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:43:38.155051433 +0000 UTC m=+635.058625310" watchObservedRunningTime="2026-07-17 18:43:38.155664054 +0000 UTC m=+635.059237931" Jul 17 18:43:38.376929 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.376892 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr"] Jul 17 18:43:38.380431 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.380406 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:43:38.383835 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.383813 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-sgwjm\"" Jul 17 18:43:38.393707 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.393679 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr"] Jul 17 18:43:38.465857 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.465816 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qm2\" (UniqueName: \"kubernetes.io/projected/336751a7-c637-4e42-988f-0c45673991b5-kube-api-access-28qm2\") pod \"limitador-operator-controller-manager-859d7b78bd-kb2nr\" (UID: \"336751a7-c637-4e42-988f-0c45673991b5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:43:38.567139 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.567098 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qm2\" (UniqueName: \"kubernetes.io/projected/336751a7-c637-4e42-988f-0c45673991b5-kube-api-access-28qm2\") pod \"limitador-operator-controller-manager-859d7b78bd-kb2nr\" (UID: \"336751a7-c637-4e42-988f-0c45673991b5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:43:38.576191 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.576162 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qm2\" (UniqueName: \"kubernetes.io/projected/336751a7-c637-4e42-988f-0c45673991b5-kube-api-access-28qm2\") pod \"limitador-operator-controller-manager-859d7b78bd-kb2nr\" (UID: \"336751a7-c637-4e42-988f-0c45673991b5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:43:38.691571 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.691532 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:43:38.859172 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:43:38.859090 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336751a7_c637_4e42_988f_0c45673991b5.slice/crio-6eb9e7470879ba950d680d7249c124fd5133e0fcbf1379eef1146417a28399a0 WatchSource:0}: Error finding container 6eb9e7470879ba950d680d7249c124fd5133e0fcbf1379eef1146417a28399a0: Status 404 returned error can't find the container with id 6eb9e7470879ba950d680d7249c124fd5133e0fcbf1379eef1146417a28399a0 Jul 17 18:43:38.861699 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:38.861675 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr"] Jul 17 18:43:39.144560 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:39.144464 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" event={"ID":"336751a7-c637-4e42-988f-0c45673991b5","Type":"ContainerStarted","Data":"6eb9e7470879ba950d680d7249c124fd5133e0fcbf1379eef1146417a28399a0"} Jul 17 18:43:41.156206 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:41.156167 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" event={"ID":"336751a7-c637-4e42-988f-0c45673991b5","Type":"ContainerStarted","Data":"555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60"} Jul 17 18:43:41.156657 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:41.156243 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:43:41.176710 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:41.176630 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" podStartSLOduration=1.327438199 podStartE2EDuration="3.176611696s" totalImagesPullingTime="1.849173497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:43:41.175886552 +0000 UTC m=+638.079460428" watchObservedRunningTime="2026-07-17 18:43:41.176611696 +0000 UTC m=+638.080185571" Jul 17 18:43:42.339979 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:42.339939 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7hd27"] Jul 17 18:43:42.343521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:42.343500 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7hd27" Jul 17 18:43:42.346203 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:42.346179 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-c7tsp\"" Jul 17 18:43:42.360283 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:42.360248 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7hd27"] Jul 17 18:43:42.505793 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:42.505752 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxkgm\" (UniqueName: \"kubernetes.io/projected/4ed78ed4-deb8-4da4-b63e-d520146d5d87-kube-api-access-sxkgm\") pod \"authorino-operator-75cc8d5cb7-7hd27\" (UID: \"4ed78ed4-deb8-4da4-b63e-d520146d5d87\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7hd27" Jul 17 18:43:42.606785 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:42.606672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxkgm\" (UniqueName: \"kubernetes.io/projected/4ed78ed4-deb8-4da4-b63e-d520146d5d87-kube-api-access-sxkgm\") pod \"authorino-operator-75cc8d5cb7-7hd27\" (UID: \"4ed78ed4-deb8-4da4-b63e-d520146d5d87\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7hd27" Jul 17 18:43:42.623163 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:42.623124 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxkgm\" (UniqueName: \"kubernetes.io/projected/4ed78ed4-deb8-4da4-b63e-d520146d5d87-kube-api-access-sxkgm\") pod \"authorino-operator-75cc8d5cb7-7hd27\" (UID: \"4ed78ed4-deb8-4da4-b63e-d520146d5d87\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7hd27" Jul 17 18:43:42.654064 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:42.654030 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7hd27" Jul 17 18:43:42.812561 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:42.812529 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7hd27"] Jul 17 18:43:42.815409 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:43:42.815370 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed78ed4_deb8_4da4_b63e_d520146d5d87.slice/crio-036a927f6884397440a87afdba1ff82428684133aa8af5b4ee042237449906dc WatchSource:0}: Error finding container 036a927f6884397440a87afdba1ff82428684133aa8af5b4ee042237449906dc: Status 404 returned error can't find the container with id 036a927f6884397440a87afdba1ff82428684133aa8af5b4ee042237449906dc Jul 17 18:43:43.167386 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:43.167347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7hd27" event={"ID":"4ed78ed4-deb8-4da4-b63e-d520146d5d87","Type":"ContainerStarted","Data":"036a927f6884397440a87afdba1ff82428684133aa8af5b4ee042237449906dc"} Jul 17 18:43:46.192776 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:46.192734 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7hd27" event={"ID":"4ed78ed4-deb8-4da4-b63e-d520146d5d87","Type":"ContainerStarted","Data":"57c5a45a18110e150b1adc798aecb72dd373c319633ab14debd9bc33b7ca695d"} Jul 17 18:43:46.193287 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:46.192832 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7hd27" Jul 17 18:43:46.218622 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:46.218566 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7hd27" podStartSLOduration=1.149407186 podStartE2EDuration="4.218549902s" totalImagesPullingTime="3.069142716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:43:46.217543057 +0000 UTC m=+643.121116933" watchObservedRunningTime="2026-07-17 18:43:46.218549902 +0000 UTC m=+643.122123777" Jul 17 18:43:47.609769 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.609727 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8"] Jul 17 18:43:47.613591 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.613563 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:43:47.616702 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.616662 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jb9j9\"" Jul 17 18:43:47.629108 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.629072 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8"] Jul 17 18:43:47.757559 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.757521 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00b92347-4b9d-44b7-840e-29eac9bae534-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" (UID: \"00b92347-4b9d-44b7-840e-29eac9bae534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:43:47.757742 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.757586 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2w4k\" (UniqueName: \"kubernetes.io/projected/00b92347-4b9d-44b7-840e-29eac9bae534-kube-api-access-v2w4k\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" (UID: \"00b92347-4b9d-44b7-840e-29eac9bae534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:43:47.858705 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.858661 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00b92347-4b9d-44b7-840e-29eac9bae534-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" (UID: \"00b92347-4b9d-44b7-840e-29eac9bae534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:43:47.858705 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.858704 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2w4k\" (UniqueName: \"kubernetes.io/projected/00b92347-4b9d-44b7-840e-29eac9bae534-kube-api-access-v2w4k\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" (UID: \"00b92347-4b9d-44b7-840e-29eac9bae534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:43:47.859044 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.859025 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00b92347-4b9d-44b7-840e-29eac9bae534-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" (UID: \"00b92347-4b9d-44b7-840e-29eac9bae534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:43:47.880500 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.880402 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2w4k\" (UniqueName: \"kubernetes.io/projected/00b92347-4b9d-44b7-840e-29eac9bae534-kube-api-access-v2w4k\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" (UID: \"00b92347-4b9d-44b7-840e-29eac9bae534\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:43:47.925363 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:47.925325 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:43:48.085812 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:43:48.085766 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b92347_4b9d_44b7_840e_29eac9bae534.slice/crio-62678ef1238cb39f13870f24345d6fb93e293368fd643f15ab15008f89228b22 WatchSource:0}: Error finding container 62678ef1238cb39f13870f24345d6fb93e293368fd643f15ab15008f89228b22: Status 404 returned error can't find the container with id 62678ef1238cb39f13870f24345d6fb93e293368fd643f15ab15008f89228b22 Jul 17 18:43:48.086939 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:48.086914 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8"] Jul 17 18:43:48.204548 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:48.204513 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" event={"ID":"00b92347-4b9d-44b7-840e-29eac9bae534","Type":"ContainerStarted","Data":"62678ef1238cb39f13870f24345d6fb93e293368fd643f15ab15008f89228b22"} Jul 17 18:43:49.146944 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:49.146907 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hlb98" Jul 17 18:43:52.164310 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:52.164275 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:43:54.239332 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:54.239299 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" event={"ID":"00b92347-4b9d-44b7-840e-29eac9bae534","Type":"ContainerStarted","Data":"a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc"} Jul 17 18:43:54.239751 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:54.239474 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:43:54.270180 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:54.270125 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" podStartSLOduration=2.105934577 podStartE2EDuration="7.270102116s" totalImagesPullingTime="5.164167539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:43:54.268254584 +0000 UTC m=+651.171828494" watchObservedRunningTime="2026-07-17 18:43:54.270102116 +0000 UTC m=+651.173676078" Jul 17 18:43:57.200370 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:43:57.200335 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7hd27" Jul 17 18:44:05.247417 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:05.247385 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:06.222575 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.222537 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822"] Jul 17 18:44:06.227260 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.227236 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:06.244953 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.244917 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822"] Jul 17 18:44:06.312893 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.312848 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq56w\" (UniqueName: \"kubernetes.io/projected/3b528d64-3565-4848-b559-a945b774bce8-kube-api-access-tq56w\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gp822\" (UID: \"3b528d64-3565-4848-b559-a945b774bce8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:06.313382 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.312976 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b528d64-3565-4848-b559-a945b774bce8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gp822\" (UID: \"3b528d64-3565-4848-b559-a945b774bce8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:06.414040 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.414002 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b528d64-3565-4848-b559-a945b774bce8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gp822\" (UID: \"3b528d64-3565-4848-b559-a945b774bce8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:06.414232 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.414062 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq56w\" (UniqueName: \"kubernetes.io/projected/3b528d64-3565-4848-b559-a945b774bce8-kube-api-access-tq56w\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gp822\" (UID: \"3b528d64-3565-4848-b559-a945b774bce8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:06.414402 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.414380 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b528d64-3565-4848-b559-a945b774bce8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gp822\" (UID: \"3b528d64-3565-4848-b559-a945b774bce8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:06.426359 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.426335 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq56w\" (UniqueName: \"kubernetes.io/projected/3b528d64-3565-4848-b559-a945b774bce8-kube-api-access-tq56w\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-gp822\" (UID: \"3b528d64-3565-4848-b559-a945b774bce8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:06.537993 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.537893 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:06.697854 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.697825 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822"] Jul 17 18:44:06.700908 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:44:06.700868 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b528d64_3565_4848_b559_a945b774bce8.slice/crio-6738654380c3969da46ad331516cad627547984d9ffa89decd7cb4514a4a7f6c WatchSource:0}: Error finding container 6738654380c3969da46ad331516cad627547984d9ffa89decd7cb4514a4a7f6c: Status 404 returned error can't find the container with id 6738654380c3969da46ad331516cad627547984d9ffa89decd7cb4514a4a7f6c Jul 17 18:44:06.913509 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.913433 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822"] Jul 17 18:44:06.923331 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.923293 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822"] Jul 17 18:44:06.935085 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.935039 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8"] Jul 17 18:44:06.935312 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.935279 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" containerName="manager" containerID="cri-o://a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc" gracePeriod=2 Jul 17 18:44:06.947125 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.947083 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg"] Jul 17 18:44:06.947801 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.947775 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b528d64-3565-4848-b559-a945b774bce8" containerName="manager" Jul 17 18:44:06.947801 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.947802 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b528d64-3565-4848-b559-a945b774bce8" containerName="manager" Jul 17 18:44:06.947996 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.947902 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b528d64-3565-4848-b559-a945b774bce8" containerName="manager" Jul 17 18:44:06.951429 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.951399 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:06.957695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.957653 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8"] Jul 17 18:44:06.981525 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:06.981472 2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"00b92347-4b9d-44b7-840e-29eac9bae534\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\": pods \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:06.984214 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:06.984134 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:06.986019 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.985987 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg"] Jul 17 18:44:06.989132 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.989091 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr"] Jul 17 18:44:06.989408 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:06.989376 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" podUID="336751a7-c637-4e42-988f-0c45673991b5" containerName="manager" containerID="cri-o://555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60" gracePeriod=2 Jul 17 18:44:07.003118 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.003082 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6"] Jul 17 18:44:07.003963 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.003898 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" containerName="manager" Jul 17 18:44:07.003963 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.003926 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" containerName="manager" Jul 17 18:44:07.004170 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.004021 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" containerName="manager" Jul 17 18:44:07.007733 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.007696 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr"] Jul 17 18:44:07.007902 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.007845 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:07.020028 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.019985 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f06969b-6199-4185-9a57-f8bc24806268-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7cmg\" (UID: \"2f06969b-6199-4185-9a57-f8bc24806268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:07.020222 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.020046 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2clq\" (UniqueName: \"kubernetes.io/projected/2f06969b-6199-4185-9a57-f8bc24806268-kube-api-access-d2clq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7cmg\" (UID: \"2f06969b-6199-4185-9a57-f8bc24806268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:07.031059 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.031017 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:07.034510 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.034426 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6"] Jul 17 18:44:07.036300 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.036263 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz"] Jul 17 18:44:07.036773 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.036755 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336751a7-c637-4e42-988f-0c45673991b5" containerName="manager" Jul 17 18:44:07.036773 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.036774 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="336751a7-c637-4e42-988f-0c45673991b5" containerName="manager" Jul 17 18:44:07.036913 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.036860 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="336751a7-c637-4e42-988f-0c45673991b5" containerName="manager" Jul 17 18:44:07.045768 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.045740 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz" Jul 17 18:44:07.055175 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.055143 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz"] Jul 17 18:44:07.067770 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.067730 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kb2nr\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="336751a7-c637-4e42-988f-0c45673991b5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:44:07.069710 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.069677 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kb2nr\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="336751a7-c637-4e42-988f-0c45673991b5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:44:07.072817 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.072782 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:07.120721 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.120685 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lr6q6\" (UID: \"bfb8db9f-111e-4383-9b21-ff7ba28bd97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:07.120911 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.120750 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f06969b-6199-4185-9a57-f8bc24806268-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7cmg\" (UID: \"2f06969b-6199-4185-9a57-f8bc24806268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:07.120911 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.120773 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzpcq\" (UniqueName: \"kubernetes.io/projected/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-kube-api-access-hzpcq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lr6q6\" (UID: \"bfb8db9f-111e-4383-9b21-ff7ba28bd97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:07.120911 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.120812 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2clq\" (UniqueName: \"kubernetes.io/projected/2f06969b-6199-4185-9a57-f8bc24806268-kube-api-access-d2clq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7cmg\" (UID: \"2f06969b-6199-4185-9a57-f8bc24806268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:07.120911 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.120840 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6ksf\" (UniqueName: \"kubernetes.io/projected/0f0f928b-1cc2-416d-934c-02fc6901e9f8-kube-api-access-j6ksf\") pod \"limitador-operator-controller-manager-859d7b78bd-ljdjz\" (UID: \"0f0f928b-1cc2-416d-934c-02fc6901e9f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz" Jul 17 18:44:07.121222 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.121199 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f06969b-6199-4185-9a57-f8bc24806268-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7cmg\" (UID: \"2f06969b-6199-4185-9a57-f8bc24806268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:07.131573 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.131535 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2clq\" (UniqueName: \"kubernetes.io/projected/2f06969b-6199-4185-9a57-f8bc24806268-kube-api-access-d2clq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7cmg\" (UID: \"2f06969b-6199-4185-9a57-f8bc24806268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:07.222255 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.222217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6ksf\" (UniqueName: \"kubernetes.io/projected/0f0f928b-1cc2-416d-934c-02fc6901e9f8-kube-api-access-j6ksf\") pod \"limitador-operator-controller-manager-859d7b78bd-ljdjz\" (UID: \"0f0f928b-1cc2-416d-934c-02fc6901e9f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz" Jul 17 18:44:07.222471 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.222327 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lr6q6\" (UID: \"bfb8db9f-111e-4383-9b21-ff7ba28bd97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:07.222471 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.222363 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpcq\" (UniqueName: \"kubernetes.io/projected/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-kube-api-access-hzpcq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lr6q6\" (UID: \"bfb8db9f-111e-4383-9b21-ff7ba28bd97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:07.222788 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.222767 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lr6q6\" (UID: \"bfb8db9f-111e-4383-9b21-ff7ba28bd97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:07.237196 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.237166 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:07.238775 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.238747 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzpcq\" (UniqueName: \"kubernetes.io/projected/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-kube-api-access-hzpcq\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lr6q6\" (UID: \"bfb8db9f-111e-4383-9b21-ff7ba28bd97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:07.239790 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.239766 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6ksf\" (UniqueName: \"kubernetes.io/projected/0f0f928b-1cc2-416d-934c-02fc6901e9f8-kube-api-access-j6ksf\") pod \"limitador-operator-controller-manager-859d7b78bd-ljdjz\" (UID: \"0f0f928b-1cc2-416d-934c-02fc6901e9f8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz" Jul 17 18:44:07.242813 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.242779 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kb2nr\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="336751a7-c637-4e42-988f-0c45673991b5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:44:07.244689 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.244651 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:07.245521 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.245490 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:44:07.247577 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.247543 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kb2nr\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="336751a7-c637-4e42-988f-0c45673991b5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:44:07.250077 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.250037 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:07.310820 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.310781 2563 generic.go:356] "Generic (PLEG): container finished" podID="336751a7-c637-4e42-988f-0c45673991b5" containerID="555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60" exitCode=0 Jul 17 18:44:07.310820 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.310837 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:44:07.311119 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.310875 2563 scope.go:117] "RemoveContainer" containerID="555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60" Jul 17 18:44:07.313261 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.313226 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:07.313261 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.313258 2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" container="manager" Jul 17 18:44:07.314920 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.314886 2563 generic.go:356] "Generic (PLEG): container finished" podID="00b92347-4b9d-44b7-840e-29eac9bae534" containerID="a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc" exitCode=0 Jul 17 18:44:07.315050 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.314961 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:07.315344 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.315318 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kb2nr\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="336751a7-c637-4e42-988f-0c45673991b5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:44:07.317517 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.317482 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kb2nr\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="336751a7-c637-4e42-988f-0c45673991b5" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kb2nr" Jul 17 18:44:07.320111 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.320072 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-gp822\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="3b528d64-3565-4848-b559-a945b774bce8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:07.322414 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.322381 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qjwl8\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qjwl8" Jul 17 18:44:07.322729 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.322706 2563 scope.go:117] "RemoveContainer" containerID="555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60" Jul 17 18:44:07.322893 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.322879 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2w4k\" (UniqueName: \"kubernetes.io/projected/00b92347-4b9d-44b7-840e-29eac9bae534-kube-api-access-v2w4k\") pod \"00b92347-4b9d-44b7-840e-29eac9bae534\" (UID: \"00b92347-4b9d-44b7-840e-29eac9bae534\") " Jul 17 18:44:07.322974 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.322935 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00b92347-4b9d-44b7-840e-29eac9bae534-extensions-socket-volume\") pod \"00b92347-4b9d-44b7-840e-29eac9bae534\" (UID: \"00b92347-4b9d-44b7-840e-29eac9bae534\") " Jul 17 18:44:07.323035 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.322985 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28qm2\" (UniqueName: \"kubernetes.io/projected/336751a7-c637-4e42-988f-0c45673991b5-kube-api-access-28qm2\") pod \"336751a7-c637-4e42-988f-0c45673991b5\" (UID: \"336751a7-c637-4e42-988f-0c45673991b5\") " Jul 17 18:44:07.323154 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.323125 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60\": container with ID starting with 555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60 not found: ID does not exist" containerID="555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60" Jul 17 18:44:07.323221 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.323169 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60"} err="failed to get container status \"555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60\": rpc error: code = NotFound desc = could not find container \"555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60\": container with ID starting with 555c7a8b73593aa97c4b64eed80d468dea5e765f56a4a3fd7ebba0266de70f60 not found: ID does not exist" Jul 17 18:44:07.323221 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.323193 2563 scope.go:117] "RemoveContainer" containerID="a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc" Jul 17 18:44:07.323418 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.323380 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00b92347-4b9d-44b7-840e-29eac9bae534-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "00b92347-4b9d-44b7-840e-29eac9bae534" (UID: "00b92347-4b9d-44b7-840e-29eac9bae534"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:44:07.325756 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.325720 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b92347-4b9d-44b7-840e-29eac9bae534-kube-api-access-v2w4k" (OuterVolumeSpecName: "kube-api-access-v2w4k") pod "00b92347-4b9d-44b7-840e-29eac9bae534" (UID: "00b92347-4b9d-44b7-840e-29eac9bae534"). InnerVolumeSpecName "kube-api-access-v2w4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:44:07.326403 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.326234 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336751a7-c637-4e42-988f-0c45673991b5-kube-api-access-28qm2" (OuterVolumeSpecName: "kube-api-access-28qm2") pod "336751a7-c637-4e42-988f-0c45673991b5" (UID: "336751a7-c637-4e42-988f-0c45673991b5"). InnerVolumeSpecName "kube-api-access-28qm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:44:07.333626 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.333589 2563 scope.go:117] "RemoveContainer" containerID="a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc" Jul 17 18:44:07.334011 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:07.333982 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc\": container with ID starting with a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc not found: ID does not exist" containerID="a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc" Jul 17 18:44:07.334094 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.334023 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc"} err="failed to get container status \"a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc\": rpc error: code = NotFound desc = could not find container \"a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc\": container with ID starting with a362aa066dc431c4e593cc8d6cd364119f994f4752f412f02b5af807608cc8fc not found: ID does not exist" Jul 17 18:44:07.355721 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.355668 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:07.366848 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.366816 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:07.374223 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.374187 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz" Jul 17 18:44:07.426903 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.423902 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2w4k\" (UniqueName: \"kubernetes.io/projected/00b92347-4b9d-44b7-840e-29eac9bae534-kube-api-access-v2w4k\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:44:07.426903 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.423940 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00b92347-4b9d-44b7-840e-29eac9bae534-extensions-socket-volume\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:44:07.426903 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.423957 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28qm2\" (UniqueName: \"kubernetes.io/projected/336751a7-c637-4e42-988f-0c45673991b5-kube-api-access-28qm2\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:44:07.566314 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.566282 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg"] Jul 17 18:44:07.570922 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:44:07.570886 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f06969b_6199_4185_9a57_f8bc24806268.slice/crio-82d29bc4ca52ab6c2582c2838ae7a583ffcb784a7d91047adfd2071d7687be65 WatchSource:0}: Error finding container 82d29bc4ca52ab6c2582c2838ae7a583ffcb784a7d91047adfd2071d7687be65: Status 404 returned error can't find the container with id 82d29bc4ca52ab6c2582c2838ae7a583ffcb784a7d91047adfd2071d7687be65 Jul 17 18:44:07.624250 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.624213 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00b92347-4b9d-44b7-840e-29eac9bae534" path="/var/lib/kubelet/pods/00b92347-4b9d-44b7-840e-29eac9bae534/volumes" Jul 17 18:44:07.624695 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.624674 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="336751a7-c637-4e42-988f-0c45673991b5" path="/var/lib/kubelet/pods/336751a7-c637-4e42-988f-0c45673991b5/volumes" Jul 17 18:44:07.808988 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.808958 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6"] Jul 17 18:44:07.811929 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:07.811895 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz"] Jul 17 18:44:07.812267 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:44:07.812236 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb8db9f_111e_4383_9b21_ff7ba28bd97d.slice/crio-e16ab81d8861edf5ea176e79f98a45bbd60666f805152b4862ee0cac58c0f456 WatchSource:0}: Error finding container e16ab81d8861edf5ea176e79f98a45bbd60666f805152b4862ee0cac58c0f456: Status 404 returned error can't find the container with id e16ab81d8861edf5ea176e79f98a45bbd60666f805152b4862ee0cac58c0f456 Jul 17 18:44:07.815027 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:44:07.815002 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0f928b_1cc2_416d_934c_02fc6901e9f8.slice/crio-8e068eb84dc60ddb904410d5148e182cbaceff3258a1d4554cbe6a01a41b22ad WatchSource:0}: Error finding container 8e068eb84dc60ddb904410d5148e182cbaceff3258a1d4554cbe6a01a41b22ad: Status 404 returned error can't find the container with id 8e068eb84dc60ddb904410d5148e182cbaceff3258a1d4554cbe6a01a41b22ad Jul 17 18:44:08.322612 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.322575 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" event={"ID":"bfb8db9f-111e-4383-9b21-ff7ba28bd97d","Type":"ContainerStarted","Data":"7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94"} Jul 17 18:44:08.323130 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.322643 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" event={"ID":"bfb8db9f-111e-4383-9b21-ff7ba28bd97d","Type":"ContainerStarted","Data":"e16ab81d8861edf5ea176e79f98a45bbd60666f805152b4862ee0cac58c0f456"} Jul 17 18:44:08.323130 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.322675 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:08.325251 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.325209 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" podUID="3b528d64-3565-4848-b559-a945b774bce8" containerName="manager" containerID="cri-o://e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5" gracePeriod=2 Jul 17 18:44:08.327315 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.327288 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz" event={"ID":"0f0f928b-1cc2-416d-934c-02fc6901e9f8","Type":"ContainerStarted","Data":"51cb5e84d7b19112ce2f242cc800f8e00a630f7beb8426a4fe78d67d192d3a3b"} Jul 17 18:44:08.327433 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.327338 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz" event={"ID":"0f0f928b-1cc2-416d-934c-02fc6901e9f8","Type":"ContainerStarted","Data":"8e068eb84dc60ddb904410d5148e182cbaceff3258a1d4554cbe6a01a41b22ad"} Jul 17 18:44:08.327433 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.327375 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz" Jul 17 18:44:08.329187 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.329161 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" event={"ID":"2f06969b-6199-4185-9a57-f8bc24806268","Type":"ContainerStarted","Data":"a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691"} Jul 17 18:44:08.329352 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.329209 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" event={"ID":"2f06969b-6199-4185-9a57-f8bc24806268","Type":"ContainerStarted","Data":"82d29bc4ca52ab6c2582c2838ae7a583ffcb784a7d91047adfd2071d7687be65"} Jul 17 18:44:08.329352 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.329263 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:08.344116 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.344043 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" podStartSLOduration=2.34402111 podStartE2EDuration="2.34402111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:08.342867987 +0000 UTC m=+665.246441864" watchObservedRunningTime="2026-07-17 18:44:08.34402111 +0000 UTC m=+665.247594988" Jul 17 18:44:08.368156 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.368101 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" podStartSLOduration=2.3680850749999998 podStartE2EDuration="2.368085075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:08.366552287 +0000 UTC m=+665.270126162" watchObservedRunningTime="2026-07-17 18:44:08.368085075 +0000 UTC m=+665.271658951" Jul 17 18:44:08.389749 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.389683 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz" podStartSLOduration=2.389665065 podStartE2EDuration="2.389665065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:08.389123008 +0000 UTC m=+665.292696886" watchObservedRunningTime="2026-07-17 18:44:08.389665065 +0000 UTC m=+665.293238940" Jul 17 18:44:08.587563 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.587486 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:08.589633 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:08.589607 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-gp822\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="3b528d64-3565-4848-b559-a945b774bce8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:08.632983 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.632947 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b528d64-3565-4848-b559-a945b774bce8-extensions-socket-volume\") pod \"3b528d64-3565-4848-b559-a945b774bce8\" (UID: \"3b528d64-3565-4848-b559-a945b774bce8\") " Jul 17 18:44:08.633150 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.633132 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq56w\" (UniqueName: \"kubernetes.io/projected/3b528d64-3565-4848-b559-a945b774bce8-kube-api-access-tq56w\") pod \"3b528d64-3565-4848-b559-a945b774bce8\" (UID: \"3b528d64-3565-4848-b559-a945b774bce8\") " Jul 17 18:44:08.633285 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.633263 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b528d64-3565-4848-b559-a945b774bce8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3b528d64-3565-4848-b559-a945b774bce8" (UID: "3b528d64-3565-4848-b559-a945b774bce8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:44:08.633382 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.633368 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b528d64-3565-4848-b559-a945b774bce8-extensions-socket-volume\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:44:08.635422 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.635396 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b528d64-3565-4848-b559-a945b774bce8-kube-api-access-tq56w" (OuterVolumeSpecName: "kube-api-access-tq56w") pod "3b528d64-3565-4848-b559-a945b774bce8" (UID: "3b528d64-3565-4848-b559-a945b774bce8"). InnerVolumeSpecName "kube-api-access-tq56w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:44:08.734864 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:08.734824 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq56w\" (UniqueName: \"kubernetes.io/projected/3b528d64-3565-4848-b559-a945b774bce8-kube-api-access-tq56w\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:44:09.337277 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:09.337241 2563 generic.go:356] "Generic (PLEG): container finished" podID="3b528d64-3565-4848-b559-a945b774bce8" containerID="e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5" exitCode=2 Jul 17 18:44:09.337807 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:09.337290 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:09.337807 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:09.337377 2563 scope.go:117] "RemoveContainer" containerID="e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5" Jul 17 18:44:09.339762 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:09.339730 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-gp822\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="3b528d64-3565-4848-b559-a945b774bce8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:09.347319 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:09.347293 2563 scope.go:117] "RemoveContainer" containerID="e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5" Jul 17 18:44:09.347642 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:09.347624 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5\": container with ID starting with e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5 not found: ID does not exist" containerID="e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5" Jul 17 18:44:09.347737 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:09.347684 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5"} err="failed to get container status \"e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5\": rpc error: code = NotFound desc = could not find container \"e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5\": container with ID starting with e6fd137a5acc6ec7a3db2872f72dea362547b735440c65018df387f06590a9d5 not found: ID does not exist" Jul 17 18:44:09.347810 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:09.347791 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-gp822\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="3b528d64-3565-4848-b559-a945b774bce8" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-gp822" Jul 17 18:44:09.623696 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:09.623618 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b528d64-3565-4848-b559-a945b774bce8" path="/var/lib/kubelet/pods/3b528d64-3565-4848-b559-a945b774bce8/volumes" Jul 17 18:44:19.340525 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.340495 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:19.340957 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.340575 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:19.340957 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.340726 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ljdjz" Jul 17 18:44:19.481591 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.481553 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6"] Jul 17 18:44:19.481781 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.481758 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" podUID="bfb8db9f-111e-4383-9b21-ff7ba28bd97d" containerName="manager" containerID="cri-o://7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94" gracePeriod=10 Jul 17 18:44:19.738761 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.738734 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:19.831183 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.831141 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-extensions-socket-volume\") pod \"bfb8db9f-111e-4383-9b21-ff7ba28bd97d\" (UID: \"bfb8db9f-111e-4383-9b21-ff7ba28bd97d\") " Jul 17 18:44:19.831378 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.831245 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzpcq\" (UniqueName: \"kubernetes.io/projected/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-kube-api-access-hzpcq\") pod \"bfb8db9f-111e-4383-9b21-ff7ba28bd97d\" (UID: \"bfb8db9f-111e-4383-9b21-ff7ba28bd97d\") " Jul 17 18:44:19.831631 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.831604 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bfb8db9f-111e-4383-9b21-ff7ba28bd97d" (UID: "bfb8db9f-111e-4383-9b21-ff7ba28bd97d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:44:19.833634 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.833609 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-kube-api-access-hzpcq" (OuterVolumeSpecName: "kube-api-access-hzpcq") pod "bfb8db9f-111e-4383-9b21-ff7ba28bd97d" (UID: "bfb8db9f-111e-4383-9b21-ff7ba28bd97d"). InnerVolumeSpecName "kube-api-access-hzpcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:44:19.932103 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.932064 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-extensions-socket-volume\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:44:19.932103 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.932094 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzpcq\" (UniqueName: \"kubernetes.io/projected/bfb8db9f-111e-4383-9b21-ff7ba28bd97d-kube-api-access-hzpcq\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:44:19.943511 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.943469 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5"] Jul 17 18:44:19.943991 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.943968 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb8db9f-111e-4383-9b21-ff7ba28bd97d" containerName="manager" Jul 17 18:44:19.943991 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.943990 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb8db9f-111e-4383-9b21-ff7ba28bd97d" containerName="manager" Jul 17 18:44:19.944148 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.944066 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb8db9f-111e-4383-9b21-ff7ba28bd97d" containerName="manager" Jul 17 18:44:19.946413 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.946396 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:44:19.976968 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:19.976928 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5"] Jul 17 18:44:20.033519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.033412 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4tl\" (UniqueName: \"kubernetes.io/projected/61879909-0d16-4f8f-80b8-154ac8d68b73-kube-api-access-vx4tl\") pod \"kuadrant-operator-controller-manager-86878c769f-hvvw5\" (UID: \"61879909-0d16-4f8f-80b8-154ac8d68b73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:44:20.033519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.033489 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61879909-0d16-4f8f-80b8-154ac8d68b73-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hvvw5\" (UID: \"61879909-0d16-4f8f-80b8-154ac8d68b73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:44:20.134727 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.134682 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4tl\" (UniqueName: \"kubernetes.io/projected/61879909-0d16-4f8f-80b8-154ac8d68b73-kube-api-access-vx4tl\") pod \"kuadrant-operator-controller-manager-86878c769f-hvvw5\" (UID: \"61879909-0d16-4f8f-80b8-154ac8d68b73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:44:20.134727 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.134729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61879909-0d16-4f8f-80b8-154ac8d68b73-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hvvw5\" (UID: \"61879909-0d16-4f8f-80b8-154ac8d68b73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:44:20.135116 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.135093 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61879909-0d16-4f8f-80b8-154ac8d68b73-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hvvw5\" (UID: \"61879909-0d16-4f8f-80b8-154ac8d68b73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:44:20.160632 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.160593 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4tl\" (UniqueName: \"kubernetes.io/projected/61879909-0d16-4f8f-80b8-154ac8d68b73-kube-api-access-vx4tl\") pod \"kuadrant-operator-controller-manager-86878c769f-hvvw5\" (UID: \"61879909-0d16-4f8f-80b8-154ac8d68b73\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:44:20.256873 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.256774 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:44:20.402410 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.402376 2563 generic.go:356] "Generic (PLEG): container finished" podID="bfb8db9f-111e-4383-9b21-ff7ba28bd97d" containerID="7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94" exitCode=0 Jul 17 18:44:20.402831 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.402476 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" Jul 17 18:44:20.402831 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.402481 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" event={"ID":"bfb8db9f-111e-4383-9b21-ff7ba28bd97d","Type":"ContainerDied","Data":"7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94"} Jul 17 18:44:20.402831 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.402540 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6" event={"ID":"bfb8db9f-111e-4383-9b21-ff7ba28bd97d","Type":"ContainerDied","Data":"e16ab81d8861edf5ea176e79f98a45bbd60666f805152b4862ee0cac58c0f456"} Jul 17 18:44:20.402831 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.402558 2563 scope.go:117] "RemoveContainer" containerID="7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94" Jul 17 18:44:20.412849 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.412822 2563 scope.go:117] "RemoveContainer" containerID="7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94" Jul 17 18:44:20.413192 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:20.413173 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94\": container with ID starting with 7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94 not found: ID does not exist" containerID="7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94" Jul 17 18:44:20.413255 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.413206 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94"} err="failed to get container status \"7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94\": rpc error: code = NotFound desc = could not find container \"7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94\": container with ID starting with 7babab65e00350c13ac8f3d704f7a24c19d9fe9b7497f725c2a345a62186fa94 not found: ID does not exist" Jul 17 18:44:20.423778 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.423703 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5"] Jul 17 18:44:20.426138 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:44:20.426112 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61879909_0d16_4f8f_80b8_154ac8d68b73.slice/crio-7fce5ed55eb79bcc08fea0b2e4529b0d971bf5c7c5a07e825edf2a0a5c59323d WatchSource:0}: Error finding container 7fce5ed55eb79bcc08fea0b2e4529b0d971bf5c7c5a07e825edf2a0a5c59323d: Status 404 returned error can't find the container with id 7fce5ed55eb79bcc08fea0b2e4529b0d971bf5c7c5a07e825edf2a0a5c59323d Jul 17 18:44:20.436731 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.436693 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6"] Jul 17 18:44:20.446188 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:20.446159 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lr6q6"] Jul 17 18:44:21.410215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:21.410178 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" event={"ID":"61879909-0d16-4f8f-80b8-154ac8d68b73","Type":"ContainerStarted","Data":"625aef2eb723afcb3976d93dc162f0977a5eab76ed1332f71211077b15a3263e"} Jul 17 18:44:21.410800 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:21.410237 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" event={"ID":"61879909-0d16-4f8f-80b8-154ac8d68b73","Type":"ContainerStarted","Data":"7fce5ed55eb79bcc08fea0b2e4529b0d971bf5c7c5a07e825edf2a0a5c59323d"} Jul 17 18:44:21.410800 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:21.410265 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:44:21.443473 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:21.443391 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" podStartSLOduration=2.443367707 podStartE2EDuration="2.443367707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:21.439722122 +0000 UTC m=+678.343295998" watchObservedRunningTime="2026-07-17 18:44:21.443367707 +0000 UTC m=+678.346941617" Jul 17 18:44:21.623209 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:21.623172 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb8db9f-111e-4383-9b21-ff7ba28bd97d" path="/var/lib/kubelet/pods/bfb8db9f-111e-4383-9b21-ff7ba28bd97d/volumes" Jul 17 18:44:32.418646 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:32.418613 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:44:32.473117 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:32.473081 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg"] Jul 17 18:44:32.473317 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:32.473291 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" podUID="2f06969b-6199-4185-9a57-f8bc24806268" containerName="manager" containerID="cri-o://a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691" gracePeriod=10 Jul 17 18:44:32.736197 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:32.736162 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:32.853381 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:32.853325 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2clq\" (UniqueName: \"kubernetes.io/projected/2f06969b-6199-4185-9a57-f8bc24806268-kube-api-access-d2clq\") pod \"2f06969b-6199-4185-9a57-f8bc24806268\" (UID: \"2f06969b-6199-4185-9a57-f8bc24806268\") " Jul 17 18:44:32.853381 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:32.853397 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f06969b-6199-4185-9a57-f8bc24806268-extensions-socket-volume\") pod \"2f06969b-6199-4185-9a57-f8bc24806268\" (UID: \"2f06969b-6199-4185-9a57-f8bc24806268\") " Jul 17 18:44:32.853846 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:32.853819 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f06969b-6199-4185-9a57-f8bc24806268-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2f06969b-6199-4185-9a57-f8bc24806268" (UID: "2f06969b-6199-4185-9a57-f8bc24806268"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:44:32.855784 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:32.855749 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f06969b-6199-4185-9a57-f8bc24806268-kube-api-access-d2clq" (OuterVolumeSpecName: "kube-api-access-d2clq") pod "2f06969b-6199-4185-9a57-f8bc24806268" (UID: "2f06969b-6199-4185-9a57-f8bc24806268"). InnerVolumeSpecName "kube-api-access-d2clq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:44:32.954489 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:32.954398 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2clq\" (UniqueName: \"kubernetes.io/projected/2f06969b-6199-4185-9a57-f8bc24806268-kube-api-access-d2clq\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:44:32.954489 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:32.954484 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f06969b-6199-4185-9a57-f8bc24806268-extensions-socket-volume\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:44:33.478099 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:33.478068 2563 generic.go:356] "Generic (PLEG): container finished" podID="2f06969b-6199-4185-9a57-f8bc24806268" containerID="a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691" exitCode=0 Jul 17 18:44:33.478537 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:33.478121 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" Jul 17 18:44:33.478537 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:33.478157 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" event={"ID":"2f06969b-6199-4185-9a57-f8bc24806268","Type":"ContainerDied","Data":"a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691"} Jul 17 18:44:33.478537 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:33.478213 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg" event={"ID":"2f06969b-6199-4185-9a57-f8bc24806268","Type":"ContainerDied","Data":"82d29bc4ca52ab6c2582c2838ae7a583ffcb784a7d91047adfd2071d7687be65"} Jul 17 18:44:33.478537 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:33.478243 2563 scope.go:117] "RemoveContainer" containerID="a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691" Jul 17 18:44:33.489244 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:33.489225 2563 scope.go:117] "RemoveContainer" containerID="a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691" Jul 17 18:44:33.489593 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:33.489571 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691\": container with ID starting with a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691 not found: ID does not exist" containerID="a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691" Jul 17 18:44:33.489649 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:33.489603 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691"} err="failed to get container status \"a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691\": rpc error: code = NotFound desc = could not find container \"a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691\": container with ID starting with a40d41bd70f97907e779224cec49a2e253ef324a5782fa31056384e70ebd4691 not found: ID does not exist" Jul 17 18:44:33.504351 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:33.504306 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg"] Jul 17 18:44:33.520078 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:33.520040 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7cmg"] Jul 17 18:44:33.622811 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:33.622779 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f06969b-6199-4185-9a57-f8bc24806268" path="/var/lib/kubelet/pods/2f06969b-6199-4185-9a57-f8bc24806268/volumes" Jul 17 18:44:35.876207 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.876174 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d"] Jul 17 18:44:35.876726 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.876580 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f06969b-6199-4185-9a57-f8bc24806268" containerName="manager" Jul 17 18:44:35.876726 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.876594 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f06969b-6199-4185-9a57-f8bc24806268" containerName="manager" Jul 17 18:44:35.876726 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.876672 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f06969b-6199-4185-9a57-f8bc24806268" containerName="manager" Jul 17 18:44:35.881820 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.881794 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:35.884230 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.884201 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6p4rf\"" Jul 17 18:44:35.891656 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.891622 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d"] Jul 17 18:44:35.982343 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.982299 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8p6n\" (UniqueName: \"kubernetes.io/projected/27310e84-6e5b-42d5-ba9a-ea4920dc6071-kube-api-access-v8p6n\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:35.982549 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.982356 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:35.982549 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.982437 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:35.982549 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.982492 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:35.982549 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.982543 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:35.982688 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.982562 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:35.982688 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.982636 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:35.982688 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.982673 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:35.982785 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:35.982693 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.083863 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.083810 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.083863 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.083861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084139 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.083986 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084139 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084037 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084139 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084139 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084130 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8p6n\" (UniqueName: \"kubernetes.io/projected/27310e84-6e5b-42d5-ba9a-ea4920dc6071-kube-api-access-v8p6n\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084346 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084192 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084346 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084234 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084346 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084272 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084346 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084329 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084602 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084558 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084663 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084615 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084663 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084636 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.084751 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.084686 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.086880 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.086844 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.086979 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.086900 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.092842 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.092816 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27310e84-6e5b-42d5-ba9a-ea4920dc6071-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.093086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.093053 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8p6n\" (UniqueName: \"kubernetes.io/projected/27310e84-6e5b-42d5-ba9a-ea4920dc6071-kube-api-access-v8p6n\") pod \"maas-default-gateway-openshift-default-56654dcf7-rvs2d\" (UID: \"27310e84-6e5b-42d5-ba9a-ea4920dc6071\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.194396 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.194355 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:36.346088 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.346063 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d"] Jul 17 18:44:36.352747 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.352708 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:44:36.352863 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.352786 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:44:36.352863 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.352822 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:44:36.500680 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.500572 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" event={"ID":"27310e84-6e5b-42d5-ba9a-ea4920dc6071","Type":"ContainerStarted","Data":"034d1079d9931b95a64088028d90e0cf0b878ee57344fdc389f344303a9eae74"} Jul 17 18:44:36.500680 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.500637 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" event={"ID":"27310e84-6e5b-42d5-ba9a-ea4920dc6071","Type":"ContainerStarted","Data":"cc94a184beba98755559b2a9d61d7214e0e5d55b0af2e9186c62966e9e5590ed"} Jul 17 18:44:36.527495 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:36.527412 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" podStartSLOduration=1.527394097 podStartE2EDuration="1.527394097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:36.525978573 +0000 UTC m=+693.429552492" watchObservedRunningTime="2026-07-17 18:44:36.527394097 +0000 UTC m=+693.430967973" Jul 17 18:44:37.195228 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:37.195192 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:37.200198 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:37.200170 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:37.507363 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:37.507279 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:37.508332 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:37.508313 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:44:51.171605 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.171562 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-689774fb5c-j5668"] Jul 17 18:44:51.697233 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.697198 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-cfk9c"] Jul 17 18:44:51.702127 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.702104 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cfk9c" Jul 17 18:44:51.704702 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.704676 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-56wj8\"" Jul 17 18:44:51.709519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.709488 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cfk9c"] Jul 17 18:44:51.728838 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.728793 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tnbw\" (UniqueName: \"kubernetes.io/projected/0879f4af-eece-4bd4-b21d-a08f3572c403-kube-api-access-4tnbw\") pod \"authorino-58cd955675-cfk9c\" (UID: \"0879f4af-eece-4bd4-b21d-a08f3572c403\") " pod="kuadrant-system/authorino-58cd955675-cfk9c" Jul 17 18:44:51.830149 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.830110 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tnbw\" (UniqueName: \"kubernetes.io/projected/0879f4af-eece-4bd4-b21d-a08f3572c403-kube-api-access-4tnbw\") pod \"authorino-58cd955675-cfk9c\" (UID: \"0879f4af-eece-4bd4-b21d-a08f3572c403\") " pod="kuadrant-system/authorino-58cd955675-cfk9c" Jul 17 18:44:51.840106 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.840065 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tnbw\" (UniqueName: \"kubernetes.io/projected/0879f4af-eece-4bd4-b21d-a08f3572c403-kube-api-access-4tnbw\") pod \"authorino-58cd955675-cfk9c\" (UID: \"0879f4af-eece-4bd4-b21d-a08f3572c403\") " pod="kuadrant-system/authorino-58cd955675-cfk9c" Jul 17 18:44:51.855552 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.855517 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqtpm"] Jul 17 18:44:51.859236 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.859211 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vqtpm" Jul 17 18:44:51.869471 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.869409 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqtpm"] Jul 17 18:44:51.930925 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:51.930883 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xt94\" (UniqueName: \"kubernetes.io/projected/ffa3501c-4d21-47aa-8852-6f3c4f8a58a2-kube-api-access-8xt94\") pod \"authorino-75d556bd77-vqtpm\" (UID: \"ffa3501c-4d21-47aa-8852-6f3c4f8a58a2\") " pod="kuadrant-system/authorino-75d556bd77-vqtpm" Jul 17 18:44:52.013927 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:52.013850 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cfk9c" Jul 17 18:44:52.032415 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:52.032373 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xt94\" (UniqueName: \"kubernetes.io/projected/ffa3501c-4d21-47aa-8852-6f3c4f8a58a2-kube-api-access-8xt94\") pod \"authorino-75d556bd77-vqtpm\" (UID: \"ffa3501c-4d21-47aa-8852-6f3c4f8a58a2\") " pod="kuadrant-system/authorino-75d556bd77-vqtpm" Jul 17 18:44:52.042556 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:52.042527 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xt94\" (UniqueName: \"kubernetes.io/projected/ffa3501c-4d21-47aa-8852-6f3c4f8a58a2-kube-api-access-8xt94\") pod \"authorino-75d556bd77-vqtpm\" (UID: \"ffa3501c-4d21-47aa-8852-6f3c4f8a58a2\") " pod="kuadrant-system/authorino-75d556bd77-vqtpm" Jul 17 18:44:52.165076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:52.164873 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cfk9c"] Jul 17 18:44:52.168152 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:44:52.168118 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0879f4af_eece_4bd4_b21d_a08f3572c403.slice/crio-85234b2ac97bc2949b1ddfcd8058fa00e25d8c06f1c4ab1e62ab1440f2a855db WatchSource:0}: Error finding container 85234b2ac97bc2949b1ddfcd8058fa00e25d8c06f1c4ab1e62ab1440f2a855db: Status 404 returned error can't find the container with id 85234b2ac97bc2949b1ddfcd8058fa00e25d8c06f1c4ab1e62ab1440f2a855db Jul 17 18:44:52.169723 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:52.169697 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vqtpm" Jul 17 18:44:52.325997 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:52.325970 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqtpm"] Jul 17 18:44:52.328700 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:44:52.328669 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffa3501c_4d21_47aa_8852_6f3c4f8a58a2.slice/crio-e139e3a798ac5d0c04119e7ec7fb65c595ff3a1ff8a7677071aa684bd6649996 WatchSource:0}: Error finding container e139e3a798ac5d0c04119e7ec7fb65c595ff3a1ff8a7677071aa684bd6649996: Status 404 returned error can't find the container with id e139e3a798ac5d0c04119e7ec7fb65c595ff3a1ff8a7677071aa684bd6649996 Jul 17 18:44:52.590538 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:52.590429 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vqtpm" event={"ID":"ffa3501c-4d21-47aa-8852-6f3c4f8a58a2","Type":"ContainerStarted","Data":"e139e3a798ac5d0c04119e7ec7fb65c595ff3a1ff8a7677071aa684bd6649996"} Jul 17 18:44:52.591733 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:52.591704 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cfk9c" event={"ID":"0879f4af-eece-4bd4-b21d-a08f3572c403","Type":"ContainerStarted","Data":"85234b2ac97bc2949b1ddfcd8058fa00e25d8c06f1c4ab1e62ab1440f2a855db"} Jul 17 18:44:56.623144 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:56.623106 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vqtpm" event={"ID":"ffa3501c-4d21-47aa-8852-6f3c4f8a58a2","Type":"ContainerStarted","Data":"b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f"} Jul 17 18:44:56.624693 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:56.624666 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cfk9c" event={"ID":"0879f4af-eece-4bd4-b21d-a08f3572c403","Type":"ContainerStarted","Data":"cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c"} Jul 17 18:44:56.643513 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:56.643441 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-vqtpm" podStartSLOduration=2.151445288 podStartE2EDuration="5.64342362s" totalImagesPullingTime="3.491978332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:56.642077765 +0000 UTC m=+713.545651642" watchObservedRunningTime="2026-07-17 18:44:56.64342362 +0000 UTC m=+713.546997496" Jul 17 18:44:56.659223 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:56.659172 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-cfk9c" podStartSLOduration=2.018832295 podStartE2EDuration="5.659151122s" totalImagesPullingTime="3.640318827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:44:56.657623909 +0000 UTC m=+713.561197785" watchObservedRunningTime="2026-07-17 18:44:56.659151122 +0000 UTC m=+713.562725052" Jul 17 18:44:56.681156 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:56.681116 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cfk9c"] Jul 17 18:44:58.634890 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:58.634834 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-cfk9c" podUID="0879f4af-eece-4bd4-b21d-a08f3572c403" containerName="authorino" containerID="cri-o://cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c" gracePeriod=30 Jul 17 18:44:58.912295 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:58.912263 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cfk9c" Jul 17 18:44:58.998059 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:58.998022 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tnbw\" (UniqueName: \"kubernetes.io/projected/0879f4af-eece-4bd4-b21d-a08f3572c403-kube-api-access-4tnbw\") pod \"0879f4af-eece-4bd4-b21d-a08f3572c403\" (UID: \"0879f4af-eece-4bd4-b21d-a08f3572c403\") " Jul 17 18:44:59.000275 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.000240 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0879f4af-eece-4bd4-b21d-a08f3572c403-kube-api-access-4tnbw" (OuterVolumeSpecName: "kube-api-access-4tnbw") pod "0879f4af-eece-4bd4-b21d-a08f3572c403" (UID: "0879f4af-eece-4bd4-b21d-a08f3572c403"). InnerVolumeSpecName "kube-api-access-4tnbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:44:59.098717 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.098675 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tnbw\" (UniqueName: \"kubernetes.io/projected/0879f4af-eece-4bd4-b21d-a08f3572c403-kube-api-access-4tnbw\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:44:59.642671 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.642634 2563 generic.go:356] "Generic (PLEG): container finished" podID="0879f4af-eece-4bd4-b21d-a08f3572c403" containerID="cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c" exitCode=0 Jul 17 18:44:59.642671 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.642680 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cfk9c" Jul 17 18:44:59.643171 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.642717 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cfk9c" event={"ID":"0879f4af-eece-4bd4-b21d-a08f3572c403","Type":"ContainerDied","Data":"cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c"} Jul 17 18:44:59.643171 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.642765 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cfk9c" event={"ID":"0879f4af-eece-4bd4-b21d-a08f3572c403","Type":"ContainerDied","Data":"85234b2ac97bc2949b1ddfcd8058fa00e25d8c06f1c4ab1e62ab1440f2a855db"} Jul 17 18:44:59.643171 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.642784 2563 scope.go:117] "RemoveContainer" containerID="cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c" Jul 17 18:44:59.654399 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.654377 2563 scope.go:117] "RemoveContainer" containerID="cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c" Jul 17 18:44:59.654749 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:44:59.654726 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c\": container with ID starting with cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c not found: ID does not exist" containerID="cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c" Jul 17 18:44:59.654847 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.654757 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c"} err="failed to get container status \"cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c\": rpc error: code = NotFound desc = could not find container \"cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c\": container with ID starting with cf185d73de0dd1d5d5f4488c37a6fcfe3028ad76daafe5abbaea7760d085785c not found: ID does not exist" Jul 17 18:44:59.659569 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.659539 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cfk9c"] Jul 17 18:44:59.664059 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:44:59.664035 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-cfk9c"] Jul 17 18:45:01.622923 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:01.622889 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0879f4af-eece-4bd4-b21d-a08f3572c403" path="/var/lib/kubelet/pods/0879f4af-eece-4bd4-b21d-a08f3572c403/volumes" Jul 17 18:45:16.194755 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.194695 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-689774fb5c-j5668" podUID="949f3554-ed03-4dc7-9707-c61348c2c484" containerName="console" containerID="cri-o://cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813" gracePeriod=15 Jul 17 18:45:16.453652 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.453586 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:45:16.558628 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.558584 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-serving-cert\") pod \"949f3554-ed03-4dc7-9707-c61348c2c484\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " Jul 17 18:45:16.558806 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.558690 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rh2m7\" (UniqueName: \"kubernetes.io/projected/949f3554-ed03-4dc7-9707-c61348c2c484-kube-api-access-rh2m7\") pod \"949f3554-ed03-4dc7-9707-c61348c2c484\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " Jul 17 18:45:16.558806 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.558751 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-console-config\") pod \"949f3554-ed03-4dc7-9707-c61348c2c484\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " Jul 17 18:45:16.558806 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.558774 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-service-ca\") pod \"949f3554-ed03-4dc7-9707-c61348c2c484\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " Jul 17 18:45:16.558985 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.558810 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-oauth-serving-cert\") pod \"949f3554-ed03-4dc7-9707-c61348c2c484\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " Jul 17 18:45:16.558985 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.558840 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-oauth-config\") pod \"949f3554-ed03-4dc7-9707-c61348c2c484\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " Jul 17 18:45:16.558985 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.558869 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-trusted-ca-bundle\") pod \"949f3554-ed03-4dc7-9707-c61348c2c484\" (UID: \"949f3554-ed03-4dc7-9707-c61348c2c484\") " Jul 17 18:45:16.559230 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.559175 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-console-config" (OuterVolumeSpecName: "console-config") pod "949f3554-ed03-4dc7-9707-c61348c2c484" (UID: "949f3554-ed03-4dc7-9707-c61348c2c484"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:45:16.559694 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.559248 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-service-ca" (OuterVolumeSpecName: "service-ca") pod "949f3554-ed03-4dc7-9707-c61348c2c484" (UID: "949f3554-ed03-4dc7-9707-c61348c2c484"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:45:16.559694 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.559282 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "949f3554-ed03-4dc7-9707-c61348c2c484" (UID: "949f3554-ed03-4dc7-9707-c61348c2c484"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:45:16.559694 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.559508 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "949f3554-ed03-4dc7-9707-c61348c2c484" (UID: "949f3554-ed03-4dc7-9707-c61348c2c484"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 18:45:16.560895 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.560871 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "949f3554-ed03-4dc7-9707-c61348c2c484" (UID: "949f3554-ed03-4dc7-9707-c61348c2c484"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:45:16.561041 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.561020 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949f3554-ed03-4dc7-9707-c61348c2c484-kube-api-access-rh2m7" (OuterVolumeSpecName: "kube-api-access-rh2m7") pod "949f3554-ed03-4dc7-9707-c61348c2c484" (UID: "949f3554-ed03-4dc7-9707-c61348c2c484"). InnerVolumeSpecName "kube-api-access-rh2m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:45:16.561083 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.561044 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "949f3554-ed03-4dc7-9707-c61348c2c484" (UID: "949f3554-ed03-4dc7-9707-c61348c2c484"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:45:16.660408 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.660369 2563 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-serving-cert\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:16.660408 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.660402 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh2m7\" (UniqueName: \"kubernetes.io/projected/949f3554-ed03-4dc7-9707-c61348c2c484-kube-api-access-rh2m7\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:16.660408 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.660414 2563 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-console-config\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:16.660666 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.660424 2563 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-service-ca\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:16.660666 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.660432 2563 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-oauth-serving-cert\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:16.660666 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.660440 2563 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/949f3554-ed03-4dc7-9707-c61348c2c484-console-oauth-config\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:16.660666 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.660475 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/949f3554-ed03-4dc7-9707-c61348c2c484-trusted-ca-bundle\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:16.732862 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.732772 2563 generic.go:356] "Generic (PLEG): container finished" podID="949f3554-ed03-4dc7-9707-c61348c2c484" containerID="cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813" exitCode=0 Jul 17 18:45:16.732862 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.732820 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689774fb5c-j5668" Jul 17 18:45:16.733089 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.732855 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689774fb5c-j5668" event={"ID":"949f3554-ed03-4dc7-9707-c61348c2c484","Type":"ContainerDied","Data":"cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813"} Jul 17 18:45:16.733089 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.732915 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689774fb5c-j5668" event={"ID":"949f3554-ed03-4dc7-9707-c61348c2c484","Type":"ContainerDied","Data":"8f42decd03818125009abc55d7e3fc4041a4ae7573ad7a7f7cbb60b5982f5cf4"} Jul 17 18:45:16.733089 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.732941 2563 scope.go:117] "RemoveContainer" containerID="cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813" Jul 17 18:45:16.742649 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.742625 2563 scope.go:117] "RemoveContainer" containerID="cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813" Jul 17 18:45:16.742968 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:45:16.742949 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813\": container with ID starting with cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813 not found: ID does not exist" containerID="cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813" Jul 17 18:45:16.743022 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.742979 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813"} err="failed to get container status \"cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813\": rpc error: code = NotFound desc = could not find container \"cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813\": container with ID starting with cb9eb7c5951a605869b9c49f2e8766cc6a823ef5273fb28d94063b33fede0813 not found: ID does not exist" Jul 17 18:45:16.753651 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.753620 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-689774fb5c-j5668"] Jul 17 18:45:16.757972 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:16.757942 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-689774fb5c-j5668"] Jul 17 18:45:17.624053 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:17.624015 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="949f3554-ed03-4dc7-9707-c61348c2c484" path="/var/lib/kubelet/pods/949f3554-ed03-4dc7-9707-c61348c2c484/volumes" Jul 17 18:45:21.810535 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:21.810495 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-n9wwk"] Jul 17 18:45:21.810927 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:21.810903 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="949f3554-ed03-4dc7-9707-c61348c2c484" containerName="console" Jul 17 18:45:21.810927 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:21.810915 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="949f3554-ed03-4dc7-9707-c61348c2c484" containerName="console" Jul 17 18:45:21.810927 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:21.810925 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0879f4af-eece-4bd4-b21d-a08f3572c403" containerName="authorino" Jul 17 18:45:21.811015 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:21.810930 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0879f4af-eece-4bd4-b21d-a08f3572c403" containerName="authorino" Jul 17 18:45:21.811015 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:21.811001 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="949f3554-ed03-4dc7-9707-c61348c2c484" containerName="console" Jul 17 18:45:21.811015 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:21.811009 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0879f4af-eece-4bd4-b21d-a08f3572c403" containerName="authorino" Jul 17 18:45:21.815348 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:21.815320 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-n9wwk" Jul 17 18:45:21.822923 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:21.822892 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-n9wwk"] Jul 17 18:45:21.907334 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:21.907294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frx9\" (UniqueName: \"kubernetes.io/projected/e38815a1-9cb9-4fcf-8af8-8c8055e11bbb-kube-api-access-4frx9\") pod \"authorino-5d4b67c446-n9wwk\" (UID: \"e38815a1-9cb9-4fcf-8af8-8c8055e11bbb\") " pod="kuadrant-system/authorino-5d4b67c446-n9wwk" Jul 17 18:45:22.008808 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.008768 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frx9\" (UniqueName: \"kubernetes.io/projected/e38815a1-9cb9-4fcf-8af8-8c8055e11bbb-kube-api-access-4frx9\") pod \"authorino-5d4b67c446-n9wwk\" (UID: \"e38815a1-9cb9-4fcf-8af8-8c8055e11bbb\") " pod="kuadrant-system/authorino-5d4b67c446-n9wwk" Jul 17 18:45:22.017034 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.017002 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frx9\" (UniqueName: \"kubernetes.io/projected/e38815a1-9cb9-4fcf-8af8-8c8055e11bbb-kube-api-access-4frx9\") pod \"authorino-5d4b67c446-n9wwk\" (UID: \"e38815a1-9cb9-4fcf-8af8-8c8055e11bbb\") " pod="kuadrant-system/authorino-5d4b67c446-n9wwk" Jul 17 18:45:22.039495 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.039462 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-n9wwk"] Jul 17 18:45:22.039751 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.039740 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-n9wwk" Jul 17 18:45:22.070845 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.070759 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f7d9db677-8h8sl"] Jul 17 18:45:22.074565 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.074537 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7d9db677-8h8sl" Jul 17 18:45:22.080322 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.080294 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f7d9db677-8h8sl"] Jul 17 18:45:22.191072 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.190934 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-n9wwk"] Jul 17 18:45:22.193531 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:45:22.193502 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode38815a1_9cb9_4fcf_8af8_8c8055e11bbb.slice/crio-35221976cdec856673c6e8500525dfb53110cba6cf7394211fd4e6779c0cdb91 WatchSource:0}: Error finding container 35221976cdec856673c6e8500525dfb53110cba6cf7394211fd4e6779c0cdb91: Status 404 returned error can't find the container with id 35221976cdec856673c6e8500525dfb53110cba6cf7394211fd4e6779c0cdb91 Jul 17 18:45:22.211055 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.211028 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vtr\" (UniqueName: \"kubernetes.io/projected/f5e3cb59-1cc9-43ba-8842-2c9b36876bdd-kube-api-access-h6vtr\") pod \"authorino-7f7d9db677-8h8sl\" (UID: \"f5e3cb59-1cc9-43ba-8842-2c9b36876bdd\") " pod="kuadrant-system/authorino-7f7d9db677-8h8sl" Jul 17 18:45:22.275781 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.275744 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f7d9db677-8h8sl"] Jul 17 18:45:22.276040 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:45:22.276022 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-h6vtr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7f7d9db677-8h8sl" podUID="f5e3cb59-1cc9-43ba-8842-2c9b36876bdd" Jul 17 18:45:22.311667 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.311638 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75dc8b6b65-6brv9"] Jul 17 18:45:22.311828 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.311727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vtr\" (UniqueName: \"kubernetes.io/projected/f5e3cb59-1cc9-43ba-8842-2c9b36876bdd-kube-api-access-h6vtr\") pod \"authorino-7f7d9db677-8h8sl\" (UID: \"f5e3cb59-1cc9-43ba-8842-2c9b36876bdd\") " pod="kuadrant-system/authorino-7f7d9db677-8h8sl" Jul 17 18:45:22.314322 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.314304 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75dc8b6b65-6brv9" Jul 17 18:45:22.316609 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.316593 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 17 18:45:22.326717 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.326695 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75dc8b6b65-6brv9"] Jul 17 18:45:22.327987 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.327962 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vtr\" (UniqueName: \"kubernetes.io/projected/f5e3cb59-1cc9-43ba-8842-2c9b36876bdd-kube-api-access-h6vtr\") pod \"authorino-7f7d9db677-8h8sl\" (UID: \"f5e3cb59-1cc9-43ba-8842-2c9b36876bdd\") " pod="kuadrant-system/authorino-7f7d9db677-8h8sl" Jul 17 18:45:22.412565 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.412531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-tls-cert\") pod \"authorino-75dc8b6b65-6brv9\" (UID: \"9abcde45-c20a-4b5e-a4ee-5c61b621ff36\") " pod="kuadrant-system/authorino-75dc8b6b65-6brv9" Jul 17 18:45:22.412565 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.412570 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbc7\" (UniqueName: \"kubernetes.io/projected/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-kube-api-access-dbbc7\") pod \"authorino-75dc8b6b65-6brv9\" (UID: \"9abcde45-c20a-4b5e-a4ee-5c61b621ff36\") " pod="kuadrant-system/authorino-75dc8b6b65-6brv9" Jul 17 18:45:22.513438 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.513405 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-tls-cert\") pod \"authorino-75dc8b6b65-6brv9\" (UID: \"9abcde45-c20a-4b5e-a4ee-5c61b621ff36\") " pod="kuadrant-system/authorino-75dc8b6b65-6brv9" Jul 17 18:45:22.513438 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.513440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbc7\" (UniqueName: \"kubernetes.io/projected/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-kube-api-access-dbbc7\") pod \"authorino-75dc8b6b65-6brv9\" (UID: \"9abcde45-c20a-4b5e-a4ee-5c61b621ff36\") " pod="kuadrant-system/authorino-75dc8b6b65-6brv9" Jul 17 18:45:22.516139 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.516118 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-tls-cert\") pod \"authorino-75dc8b6b65-6brv9\" (UID: \"9abcde45-c20a-4b5e-a4ee-5c61b621ff36\") " pod="kuadrant-system/authorino-75dc8b6b65-6brv9" Jul 17 18:45:22.521550 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.521527 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbc7\" (UniqueName: \"kubernetes.io/projected/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-kube-api-access-dbbc7\") pod \"authorino-75dc8b6b65-6brv9\" (UID: \"9abcde45-c20a-4b5e-a4ee-5c61b621ff36\") " pod="kuadrant-system/authorino-75dc8b6b65-6brv9" Jul 17 18:45:22.624260 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.624230 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75dc8b6b65-6brv9" Jul 17 18:45:22.768533 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.768486 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7d9db677-8h8sl" Jul 17 18:45:22.768533 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.768497 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-n9wwk" event={"ID":"e38815a1-9cb9-4fcf-8af8-8c8055e11bbb","Type":"ContainerStarted","Data":"bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f"} Jul 17 18:45:22.768533 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.768550 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-n9wwk" event={"ID":"e38815a1-9cb9-4fcf-8af8-8c8055e11bbb","Type":"ContainerStarted","Data":"35221976cdec856673c6e8500525dfb53110cba6cf7394211fd4e6779c0cdb91"} Jul 17 18:45:22.768865 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.768548 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-n9wwk" podUID="e38815a1-9cb9-4fcf-8af8-8c8055e11bbb" containerName="authorino" containerID="cri-o://bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f" gracePeriod=30 Jul 17 18:45:22.775902 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.775879 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7d9db677-8h8sl" Jul 17 18:45:22.785188 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.785155 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75dc8b6b65-6brv9"] Jul 17 18:45:22.786084 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.785848 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-n9wwk" podStartSLOduration=1.393832038 podStartE2EDuration="1.785829291s" totalImagesPullingTime="391.997253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:22.785661239 +0000 UTC m=+739.689235116" watchObservedRunningTime="2026-07-17 18:45:22.785829291 +0000 UTC m=+739.689403166" Jul 17 18:45:22.811288 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:45:22.811250 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9abcde45_c20a_4b5e_a4ee_5c61b621ff36.slice/crio-f06de4874a7117c31bb2d0d12ed69d4160a72720d8c1af4b98a747e2da597681 WatchSource:0}: Error finding container f06de4874a7117c31bb2d0d12ed69d4160a72720d8c1af4b98a747e2da597681: Status 404 returned error can't find the container with id f06de4874a7117c31bb2d0d12ed69d4160a72720d8c1af4b98a747e2da597681 Jul 17 18:45:22.917954 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.917914 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6vtr\" (UniqueName: \"kubernetes.io/projected/f5e3cb59-1cc9-43ba-8842-2c9b36876bdd-kube-api-access-h6vtr\") pod \"f5e3cb59-1cc9-43ba-8842-2c9b36876bdd\" (UID: \"f5e3cb59-1cc9-43ba-8842-2c9b36876bdd\") " Jul 17 18:45:22.920179 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:22.920146 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e3cb59-1cc9-43ba-8842-2c9b36876bdd-kube-api-access-h6vtr" (OuterVolumeSpecName: "kube-api-access-h6vtr") pod "f5e3cb59-1cc9-43ba-8842-2c9b36876bdd" (UID: "f5e3cb59-1cc9-43ba-8842-2c9b36876bdd"). InnerVolumeSpecName "kube-api-access-h6vtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:45:23.019258 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.019227 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6vtr\" (UniqueName: \"kubernetes.io/projected/f5e3cb59-1cc9-43ba-8842-2c9b36876bdd-kube-api-access-h6vtr\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:23.065703 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.065673 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-n9wwk" Jul 17 18:45:23.220831 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.220786 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4frx9\" (UniqueName: \"kubernetes.io/projected/e38815a1-9cb9-4fcf-8af8-8c8055e11bbb-kube-api-access-4frx9\") pod \"e38815a1-9cb9-4fcf-8af8-8c8055e11bbb\" (UID: \"e38815a1-9cb9-4fcf-8af8-8c8055e11bbb\") " Jul 17 18:45:23.227838 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.223878 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38815a1-9cb9-4fcf-8af8-8c8055e11bbb-kube-api-access-4frx9" (OuterVolumeSpecName: "kube-api-access-4frx9") pod "e38815a1-9cb9-4fcf-8af8-8c8055e11bbb" (UID: "e38815a1-9cb9-4fcf-8af8-8c8055e11bbb"). InnerVolumeSpecName "kube-api-access-4frx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:45:23.322085 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.322056 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4frx9\" (UniqueName: \"kubernetes.io/projected/e38815a1-9cb9-4fcf-8af8-8c8055e11bbb-kube-api-access-4frx9\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:23.774944 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.774908 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75dc8b6b65-6brv9" event={"ID":"9abcde45-c20a-4b5e-a4ee-5c61b621ff36","Type":"ContainerStarted","Data":"78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb"} Jul 17 18:45:23.775131 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.774956 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75dc8b6b65-6brv9" event={"ID":"9abcde45-c20a-4b5e-a4ee-5c61b621ff36","Type":"ContainerStarted","Data":"f06de4874a7117c31bb2d0d12ed69d4160a72720d8c1af4b98a747e2da597681"} Jul 17 18:45:23.776212 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.776186 2563 generic.go:356] "Generic (PLEG): container finished" podID="e38815a1-9cb9-4fcf-8af8-8c8055e11bbb" containerID="bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f" exitCode=0 Jul 17 18:45:23.776329 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.776240 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-n9wwk" Jul 17 18:45:23.776329 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.776270 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-n9wwk" event={"ID":"e38815a1-9cb9-4fcf-8af8-8c8055e11bbb","Type":"ContainerDied","Data":"bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f"} Jul 17 18:45:23.776329 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.776305 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-n9wwk" event={"ID":"e38815a1-9cb9-4fcf-8af8-8c8055e11bbb","Type":"ContainerDied","Data":"35221976cdec856673c6e8500525dfb53110cba6cf7394211fd4e6779c0cdb91"} Jul 17 18:45:23.776329 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.776321 2563 scope.go:117] "RemoveContainer" containerID="bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f" Jul 17 18:45:23.776518 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.776419 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f7d9db677-8h8sl" Jul 17 18:45:23.785237 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.785217 2563 scope.go:117] "RemoveContainer" containerID="bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f" Jul 17 18:45:23.785595 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:45:23.785575 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f\": container with ID starting with bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f not found: ID does not exist" containerID="bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f" Jul 17 18:45:23.785684 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.785609 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f"} err="failed to get container status \"bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f\": rpc error: code = NotFound desc = could not find container \"bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f\": container with ID starting with bafa868574c250325474615f6665fa65bb60c7358442c9be59347d2bd36b5f3f not found: ID does not exist" Jul 17 18:45:23.799701 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.799646 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75dc8b6b65-6brv9" podStartSLOduration=1.419612348 podStartE2EDuration="1.79962958s" totalImagesPullingTime="380.017232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:23.788357651 +0000 UTC m=+740.691931527" watchObservedRunningTime="2026-07-17 18:45:23.79962958 +0000 UTC m=+740.703203458" Jul 17 18:45:23.801086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.801051 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-n9wwk"] Jul 17 18:45:23.803331 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.803310 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-n9wwk"] Jul 17 18:45:23.811936 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.811908 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqtpm"] Jul 17 18:45:23.812380 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.812080 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-vqtpm" podUID="ffa3501c-4d21-47aa-8852-6f3c4f8a58a2" containerName="authorino" containerID="cri-o://b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f" gracePeriod=30 Jul 17 18:45:23.823324 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.823287 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f7d9db677-8h8sl"] Jul 17 18:45:23.825702 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:23.825678 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f7d9db677-8h8sl"] Jul 17 18:45:24.071627 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.071603 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vqtpm" Jul 17 18:45:24.229440 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.229342 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xt94\" (UniqueName: \"kubernetes.io/projected/ffa3501c-4d21-47aa-8852-6f3c4f8a58a2-kube-api-access-8xt94\") pod \"ffa3501c-4d21-47aa-8852-6f3c4f8a58a2\" (UID: \"ffa3501c-4d21-47aa-8852-6f3c4f8a58a2\") " Jul 17 18:45:24.231690 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.231661 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa3501c-4d21-47aa-8852-6f3c4f8a58a2-kube-api-access-8xt94" (OuterVolumeSpecName: "kube-api-access-8xt94") pod "ffa3501c-4d21-47aa-8852-6f3c4f8a58a2" (UID: "ffa3501c-4d21-47aa-8852-6f3c4f8a58a2"). InnerVolumeSpecName "kube-api-access-8xt94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:45:24.331098 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.331057 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xt94\" (UniqueName: \"kubernetes.io/projected/ffa3501c-4d21-47aa-8852-6f3c4f8a58a2-kube-api-access-8xt94\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:24.783088 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.783055 2563 generic.go:356] "Generic (PLEG): container finished" podID="ffa3501c-4d21-47aa-8852-6f3c4f8a58a2" containerID="b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f" exitCode=0 Jul 17 18:45:24.783292 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.783100 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vqtpm" Jul 17 18:45:24.783292 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.783130 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vqtpm" event={"ID":"ffa3501c-4d21-47aa-8852-6f3c4f8a58a2","Type":"ContainerDied","Data":"b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f"} Jul 17 18:45:24.783292 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.783167 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vqtpm" event={"ID":"ffa3501c-4d21-47aa-8852-6f3c4f8a58a2","Type":"ContainerDied","Data":"e139e3a798ac5d0c04119e7ec7fb65c595ff3a1ff8a7677071aa684bd6649996"} Jul 17 18:45:24.783292 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.783191 2563 scope.go:117] "RemoveContainer" containerID="b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f" Jul 17 18:45:24.792503 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.792476 2563 scope.go:117] "RemoveContainer" containerID="b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f" Jul 17 18:45:24.792849 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:45:24.792827 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f\": container with ID starting with b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f not found: ID does not exist" containerID="b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f" Jul 17 18:45:24.792905 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.792860 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f"} err="failed to get container status \"b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f\": rpc error: code = NotFound desc = could not find container \"b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f\": container with ID starting with b0e4faecafd06f6be9f446ee032e1ba1824d99004b609e697ce94d92a115c12f not found: ID does not exist" Jul 17 18:45:24.813020 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.812984 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqtpm"] Jul 17 18:45:24.817266 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:24.817234 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vqtpm"] Jul 17 18:45:25.623648 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:25.623614 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e38815a1-9cb9-4fcf-8af8-8c8055e11bbb" path="/var/lib/kubelet/pods/e38815a1-9cb9-4fcf-8af8-8c8055e11bbb/volumes" Jul 17 18:45:25.623967 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:25.623954 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e3cb59-1cc9-43ba-8842-2c9b36876bdd" path="/var/lib/kubelet/pods/f5e3cb59-1cc9-43ba-8842-2c9b36876bdd/volumes" Jul 17 18:45:25.624155 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:25.624144 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa3501c-4d21-47aa-8852-6f3c4f8a58a2" path="/var/lib/kubelet/pods/ffa3501c-4d21-47aa-8852-6f3c4f8a58a2/volumes" Jul 17 18:45:28.449419 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.449378 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b6c67555b-54r45"] Jul 17 18:45:28.449830 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.449791 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa3501c-4d21-47aa-8852-6f3c4f8a58a2" containerName="authorino" Jul 17 18:45:28.449830 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.449803 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa3501c-4d21-47aa-8852-6f3c4f8a58a2" containerName="authorino" Jul 17 18:45:28.449830 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.449814 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e38815a1-9cb9-4fcf-8af8-8c8055e11bbb" containerName="authorino" Jul 17 18:45:28.449830 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.449819 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e38815a1-9cb9-4fcf-8af8-8c8055e11bbb" containerName="authorino" Jul 17 18:45:28.449973 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.449895 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e38815a1-9cb9-4fcf-8af8-8c8055e11bbb" containerName="authorino" Jul 17 18:45:28.449973 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.449905 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa3501c-4d21-47aa-8852-6f3c4f8a58a2" containerName="authorino" Jul 17 18:45:28.453282 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.453260 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:28.455893 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.455867 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8hd7g\"" Jul 17 18:45:28.456678 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.456641 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 18:45:28.456678 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.456666 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 18:45:28.462397 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.462366 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b6c67555b-54r45"] Jul 17 18:45:28.467720 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.467690 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b980cc2-b5c6-4343-ba90-95bf7ab71239-webhook-cert\") pod \"maas-controller-6b6c67555b-54r45\" (UID: \"8b980cc2-b5c6-4343-ba90-95bf7ab71239\") " pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:28.467863 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.467829 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxm9w\" (UniqueName: \"kubernetes.io/projected/8b980cc2-b5c6-4343-ba90-95bf7ab71239-kube-api-access-fxm9w\") pod \"maas-controller-6b6c67555b-54r45\" (UID: \"8b980cc2-b5c6-4343-ba90-95bf7ab71239\") " pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:28.568340 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.568299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxm9w\" (UniqueName: \"kubernetes.io/projected/8b980cc2-b5c6-4343-ba90-95bf7ab71239-kube-api-access-fxm9w\") pod \"maas-controller-6b6c67555b-54r45\" (UID: \"8b980cc2-b5c6-4343-ba90-95bf7ab71239\") " pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:28.568550 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.568363 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b980cc2-b5c6-4343-ba90-95bf7ab71239-webhook-cert\") pod \"maas-controller-6b6c67555b-54r45\" (UID: \"8b980cc2-b5c6-4343-ba90-95bf7ab71239\") " pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:28.571026 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.571005 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b980cc2-b5c6-4343-ba90-95bf7ab71239-webhook-cert\") pod \"maas-controller-6b6c67555b-54r45\" (UID: \"8b980cc2-b5c6-4343-ba90-95bf7ab71239\") " pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:28.578331 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.578302 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxm9w\" (UniqueName: \"kubernetes.io/projected/8b980cc2-b5c6-4343-ba90-95bf7ab71239-kube-api-access-fxm9w\") pod \"maas-controller-6b6c67555b-54r45\" (UID: \"8b980cc2-b5c6-4343-ba90-95bf7ab71239\") " pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:28.765594 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.765507 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:28.908894 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:28.908872 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b6c67555b-54r45"] Jul 17 18:45:28.910991 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:45:28.910952 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b980cc2_b5c6_4343_ba90_95bf7ab71239.slice/crio-acef96e912871bb8145c77e1c5ef8fe218eb701d65543d5b3f5479063a7607a6 WatchSource:0}: Error finding container acef96e912871bb8145c77e1c5ef8fe218eb701d65543d5b3f5479063a7607a6: Status 404 returned error can't find the container with id acef96e912871bb8145c77e1c5ef8fe218eb701d65543d5b3f5479063a7607a6 Jul 17 18:45:29.822841 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:29.822793 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6c67555b-54r45" event={"ID":"8b980cc2-b5c6-4343-ba90-95bf7ab71239","Type":"ContainerStarted","Data":"acef96e912871bb8145c77e1c5ef8fe218eb701d65543d5b3f5479063a7607a6"} Jul 17 18:45:31.838216 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:31.838117 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6c67555b-54r45" event={"ID":"8b980cc2-b5c6-4343-ba90-95bf7ab71239","Type":"ContainerStarted","Data":"ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4"} Jul 17 18:45:31.838711 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:31.838220 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:31.859727 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:31.859669 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6b6c67555b-54r45" podStartSLOduration=1.210221032 podStartE2EDuration="3.859650249s" totalImagesPullingTime="2.649429217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:45:31.857224019 +0000 UTC m=+748.760797908" watchObservedRunningTime="2026-07-17 18:45:31.859650249 +0000 UTC m=+748.763224166" Jul 17 18:45:42.846530 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:42.846497 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:43.417541 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.417500 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b6c67555b-54r45"] Jul 17 18:45:43.417726 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.417697 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6b6c67555b-54r45" podUID="8b980cc2-b5c6-4343-ba90-95bf7ab71239" containerName="manager" containerID="cri-o://ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4" gracePeriod=30 Jul 17 18:45:43.675135 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.675064 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:43.682021 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.681999 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b980cc2-b5c6-4343-ba90-95bf7ab71239-webhook-cert\") pod \"8b980cc2-b5c6-4343-ba90-95bf7ab71239\" (UID: \"8b980cc2-b5c6-4343-ba90-95bf7ab71239\") " Jul 17 18:45:43.682169 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.682072 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxm9w\" (UniqueName: \"kubernetes.io/projected/8b980cc2-b5c6-4343-ba90-95bf7ab71239-kube-api-access-fxm9w\") pod \"8b980cc2-b5c6-4343-ba90-95bf7ab71239\" (UID: \"8b980cc2-b5c6-4343-ba90-95bf7ab71239\") " Jul 17 18:45:43.684405 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.684379 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b980cc2-b5c6-4343-ba90-95bf7ab71239-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8b980cc2-b5c6-4343-ba90-95bf7ab71239" (UID: "8b980cc2-b5c6-4343-ba90-95bf7ab71239"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:45:43.684548 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.684439 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b980cc2-b5c6-4343-ba90-95bf7ab71239-kube-api-access-fxm9w" (OuterVolumeSpecName: "kube-api-access-fxm9w") pod "8b980cc2-b5c6-4343-ba90-95bf7ab71239" (UID: "8b980cc2-b5c6-4343-ba90-95bf7ab71239"). InnerVolumeSpecName "kube-api-access-fxm9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:45:43.783646 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.783608 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxm9w\" (UniqueName: \"kubernetes.io/projected/8b980cc2-b5c6-4343-ba90-95bf7ab71239-kube-api-access-fxm9w\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:43.783646 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.783636 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b980cc2-b5c6-4343-ba90-95bf7ab71239-webhook-cert\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:45:43.901751 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.901717 2563 generic.go:356] "Generic (PLEG): container finished" podID="8b980cc2-b5c6-4343-ba90-95bf7ab71239" containerID="ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4" exitCode=0 Jul 17 18:45:43.902266 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.901767 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b6c67555b-54r45" Jul 17 18:45:43.902266 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.901818 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6c67555b-54r45" event={"ID":"8b980cc2-b5c6-4343-ba90-95bf7ab71239","Type":"ContainerDied","Data":"ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4"} Jul 17 18:45:43.902266 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.901878 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b6c67555b-54r45" event={"ID":"8b980cc2-b5c6-4343-ba90-95bf7ab71239","Type":"ContainerDied","Data":"acef96e912871bb8145c77e1c5ef8fe218eb701d65543d5b3f5479063a7607a6"} Jul 17 18:45:43.902266 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.901915 2563 scope.go:117] "RemoveContainer" containerID="ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4" Jul 17 18:45:43.911916 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.911897 2563 scope.go:117] "RemoveContainer" containerID="ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4" Jul 17 18:45:43.912247 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:45:43.912224 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4\": container with ID starting with ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4 not found: ID does not exist" containerID="ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4" Jul 17 18:45:43.912348 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.912258 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4"} err="failed to get container status \"ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4\": rpc error: code = NotFound desc = could not find container \"ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4\": container with ID starting with ae91e9b786a1dbee6e39a64d3bf9c8902e851001ff9d09eceb0786ac055d76c4 not found: ID does not exist" Jul 17 18:45:43.922975 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.922942 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b6c67555b-54r45"] Jul 17 18:45:43.927460 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:43.927376 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b6c67555b-54r45"] Jul 17 18:45:45.624591 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:45:45.624548 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b980cc2-b5c6-4343-ba90-95bf7ab71239" path="/var/lib/kubelet/pods/8b980cc2-b5c6-4343-ba90-95bf7ab71239/volumes" Jul 17 18:46:05.124159 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.124107 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-srdlz"] Jul 17 18:46:05.124749 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.124728 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b980cc2-b5c6-4343-ba90-95bf7ab71239" containerName="manager" Jul 17 18:46:05.124817 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.124751 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b980cc2-b5c6-4343-ba90-95bf7ab71239" containerName="manager" Jul 17 18:46:05.124874 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.124859 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b980cc2-b5c6-4343-ba90-95bf7ab71239" containerName="manager" Jul 17 18:46:05.130552 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.130521 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" Jul 17 18:46:05.133333 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.133303 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bvk8r\"" Jul 17 18:46:05.133333 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.133303 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 17 18:46:05.141313 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.141275 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-srdlz"] Jul 17 18:46:05.159161 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.159123 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-dc72j"] Jul 17 18:46:05.162103 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.162072 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" Jul 17 18:46:05.171956 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.171926 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-dc72j"] Jul 17 18:46:05.184987 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.184953 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962e09eb-d2ca-41e4-ae57-16cfb445573c-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-srdlz\" (UID: \"962e09eb-d2ca-41e4-ae57-16cfb445573c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" Jul 17 18:46:05.185219 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.184994 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq5np\" (UniqueName: \"kubernetes.io/projected/962e09eb-d2ca-41e4-ae57-16cfb445573c-kube-api-access-bq5np\") pod \"payload-pre-processing-7677b774bb-srdlz\" (UID: \"962e09eb-d2ca-41e4-ae57-16cfb445573c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" Jul 17 18:46:05.285674 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.285615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962e09eb-d2ca-41e4-ae57-16cfb445573c-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-srdlz\" (UID: \"962e09eb-d2ca-41e4-ae57-16cfb445573c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" Jul 17 18:46:05.285674 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.285676 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq5np\" (UniqueName: \"kubernetes.io/projected/962e09eb-d2ca-41e4-ae57-16cfb445573c-kube-api-access-bq5np\") pod \"payload-pre-processing-7677b774bb-srdlz\" (UID: \"962e09eb-d2ca-41e4-ae57-16cfb445573c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" Jul 17 18:46:05.285936 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.285738 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fqbc\" (UniqueName: \"kubernetes.io/projected/8e68af24-81ea-4746-9d79-3ad158296036-kube-api-access-6fqbc\") pod \"payload-processing-5bf47cd988-dc72j\" (UID: \"8e68af24-81ea-4746-9d79-3ad158296036\") " pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" Jul 17 18:46:05.285936 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.285804 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e68af24-81ea-4746-9d79-3ad158296036-plugins-config-volume\") pod \"payload-processing-5bf47cd988-dc72j\" (UID: \"8e68af24-81ea-4746-9d79-3ad158296036\") " pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" Jul 17 18:46:05.286312 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.286293 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962e09eb-d2ca-41e4-ae57-16cfb445573c-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-srdlz\" (UID: \"962e09eb-d2ca-41e4-ae57-16cfb445573c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" Jul 17 18:46:05.300726 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.300689 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq5np\" (UniqueName: \"kubernetes.io/projected/962e09eb-d2ca-41e4-ae57-16cfb445573c-kube-api-access-bq5np\") pod \"payload-pre-processing-7677b774bb-srdlz\" (UID: \"962e09eb-d2ca-41e4-ae57-16cfb445573c\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" Jul 17 18:46:05.386927 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.386813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fqbc\" (UniqueName: \"kubernetes.io/projected/8e68af24-81ea-4746-9d79-3ad158296036-kube-api-access-6fqbc\") pod \"payload-processing-5bf47cd988-dc72j\" (UID: \"8e68af24-81ea-4746-9d79-3ad158296036\") " pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" Jul 17 18:46:05.387119 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.386930 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e68af24-81ea-4746-9d79-3ad158296036-plugins-config-volume\") pod \"payload-processing-5bf47cd988-dc72j\" (UID: \"8e68af24-81ea-4746-9d79-3ad158296036\") " pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" Jul 17 18:46:05.387672 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.387650 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e68af24-81ea-4746-9d79-3ad158296036-plugins-config-volume\") pod \"payload-processing-5bf47cd988-dc72j\" (UID: \"8e68af24-81ea-4746-9d79-3ad158296036\") " pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" Jul 17 18:46:05.405209 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.405169 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fqbc\" (UniqueName: \"kubernetes.io/projected/8e68af24-81ea-4746-9d79-3ad158296036-kube-api-access-6fqbc\") pod \"payload-processing-5bf47cd988-dc72j\" (UID: \"8e68af24-81ea-4746-9d79-3ad158296036\") " pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" Jul 17 18:46:05.443091 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.443051 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" Jul 17 18:46:05.476951 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.476921 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" Jul 17 18:46:05.606222 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.606187 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-srdlz"] Jul 17 18:46:05.608526 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:46:05.608475 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962e09eb_d2ca_41e4_ae57_16cfb445573c.slice/crio-7826856dfc2c56d134f3cfe22feae984a527af58e933639659ad06e29bde4af6 WatchSource:0}: Error finding container 7826856dfc2c56d134f3cfe22feae984a527af58e933639659ad06e29bde4af6: Status 404 returned error can't find the container with id 7826856dfc2c56d134f3cfe22feae984a527af58e933639659ad06e29bde4af6 Jul 17 18:46:05.649739 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:05.649682 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-dc72j"] Jul 17 18:46:05.652437 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:46:05.652410 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e68af24_81ea_4746_9d79_3ad158296036.slice/crio-a5e3fc4a9513de9b33071a211a4a4dd4e39ab18f01506bf0d9d8d535df99126a WatchSource:0}: Error finding container a5e3fc4a9513de9b33071a211a4a4dd4e39ab18f01506bf0d9d8d535df99126a: Status 404 returned error can't find the container with id a5e3fc4a9513de9b33071a211a4a4dd4e39ab18f01506bf0d9d8d535df99126a Jul 17 18:46:06.026919 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:06.026879 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" event={"ID":"962e09eb-d2ca-41e4-ae57-16cfb445573c","Type":"ContainerStarted","Data":"7826856dfc2c56d134f3cfe22feae984a527af58e933639659ad06e29bde4af6"} Jul 17 18:46:06.028084 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:06.028059 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" event={"ID":"8e68af24-81ea-4746-9d79-3ad158296036","Type":"ContainerStarted","Data":"a5e3fc4a9513de9b33071a211a4a4dd4e39ab18f01506bf0d9d8d535df99126a"} Jul 17 18:46:08.041840 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:08.041797 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" event={"ID":"8e68af24-81ea-4746-9d79-3ad158296036","Type":"ContainerStarted","Data":"a4d16a5e53e2723b8c58ff1a06bf72925ae4f32b214a87c613acc69d80e9e7bf"} Jul 17 18:46:08.044302 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:08.044274 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" Jul 17 18:46:08.044613 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:08.044596 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" Jul 17 18:46:08.044799 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:08.044776 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" event={"ID":"962e09eb-d2ca-41e4-ae57-16cfb445573c","Type":"ContainerStarted","Data":"4e8a710f7a1b9b120387263114ab67d8c89bdf1693f806f51454e2abaf0cc147"} Jul 17 18:46:08.061580 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:08.061437 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" podStartSLOduration=0.896786659 podStartE2EDuration="3.061415374s" totalImagesPullingTime="2.164628715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:08.06034715 +0000 UTC m=+784.963921027" watchObservedRunningTime="2026-07-17 18:46:08.061415374 +0000 UTC m=+784.964989251" Jul 17 18:46:08.080583 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:08.080519 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" podStartSLOduration=0.878697813 podStartE2EDuration="3.080496509s" totalImagesPullingTime="2.201798696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:08.077344253 +0000 UTC m=+784.980918141" watchObservedRunningTime="2026-07-17 18:46:08.080496509 +0000 UTC m=+784.984070387" Jul 17 18:46:14.081128 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:14.081089 2563 generic.go:356] "Generic (PLEG): container finished" podID="27310e84-6e5b-42d5-ba9a-ea4920dc6071" containerID="034d1079d9931b95a64088028d90e0cf0b878ee57344fdc389f344303a9eae74" exitCode=0 Jul 17 18:46:14.081646 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:14.081149 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" event={"ID":"27310e84-6e5b-42d5-ba9a-ea4920dc6071","Type":"ContainerDied","Data":"034d1079d9931b95a64088028d90e0cf0b878ee57344fdc389f344303a9eae74"} Jul 17 18:46:14.081646 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:14.081568 2563 scope.go:117] "RemoveContainer" containerID="034d1079d9931b95a64088028d90e0cf0b878ee57344fdc389f344303a9eae74" Jul 17 18:46:14.082839 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:14.082727 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:46:14.082839 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:14.082790 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:46:14.082839 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:14.082829 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 18:46:15.088433 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:15.088396 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" event={"ID":"27310e84-6e5b-42d5-ba9a-ea4920dc6071","Type":"ContainerStarted","Data":"b6c6b6d0bfd23baacb74cfc9e6ac7dc9330f8c24171edb460455528069cb8d4f"} Jul 17 18:46:15.195147 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:15.195109 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:46:15.195385 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:15.195361 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rvs2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 17 18:46:15.195509 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:15.195485 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" podUID="27310e84-6e5b-42d5-ba9a-ea4920dc6071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 17 18:46:16.195771 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:16.195734 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rvs2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 17 18:46:16.196236 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:16.195800 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" podUID="27310e84-6e5b-42d5-ba9a-ea4920dc6071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 17 18:46:17.195264 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:17.195220 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rvs2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 17 18:46:17.195481 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:17.195293 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" podUID="27310e84-6e5b-42d5-ba9a-ea4920dc6071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 17 18:46:18.195187 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:18.195146 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-rvs2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 17 18:46:18.195649 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:18.195212 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" podUID="27310e84-6e5b-42d5-ba9a-ea4920dc6071" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 17 18:46:19.055101 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:19.055066 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-srdlz" Jul 17 18:46:19.198820 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:19.198790 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:46:19.199198 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:19.199174 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:46:19.199986 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:19.199963 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-rvs2d" Jul 17 18:46:20.061620 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:20.061587 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-dc72j" Jul 17 18:46:26.493015 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.492920 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd"] Jul 17 18:46:26.495893 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.495866 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.499092 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.499068 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 18:46:26.500266 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.500240 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 18:46:26.500424 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.500291 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-skht6\"" Jul 17 18:46:26.500424 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.500241 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 17 18:46:26.508311 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.508274 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd"] Jul 17 18:46:26.591886 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.591850 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.592098 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.592009 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.592098 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.592046 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmk9q\" (UniqueName: \"kubernetes.io/projected/1153df46-a624-48d4-bbd1-a3d2154479d3-kube-api-access-vmk9q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.592190 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.592101 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.592190 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.592158 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1153df46-a624-48d4-bbd1-a3d2154479d3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.592284 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.592204 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.693820 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.693774 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.694027 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.693826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmk9q\" (UniqueName: \"kubernetes.io/projected/1153df46-a624-48d4-bbd1-a3d2154479d3-kube-api-access-vmk9q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.694027 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.693867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.694027 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.693925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1153df46-a624-48d4-bbd1-a3d2154479d3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.694027 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.693956 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.694027 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.693990 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.694295 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.694244 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.694535 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.694512 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.694631 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.694581 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.696287 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.696262 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1153df46-a624-48d4-bbd1-a3d2154479d3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.696657 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.696639 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1153df46-a624-48d4-bbd1-a3d2154479d3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.701921 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.701895 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmk9q\" (UniqueName: \"kubernetes.io/projected/1153df46-a624-48d4-bbd1-a3d2154479d3-kube-api-access-vmk9q\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jpknd\" (UID: \"1153df46-a624-48d4-bbd1-a3d2154479d3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.810086 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.809985 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:26.966507 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:26.966472 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd"] Jul 17 18:46:26.969759 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:46:26.969717 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1153df46_a624_48d4_bbd1_a3d2154479d3.slice/crio-fd734ec20c3d86d1b6b140e0990f95bb5e967ff6e74dd6c6b905543b7fa62c92 WatchSource:0}: Error finding container fd734ec20c3d86d1b6b140e0990f95bb5e967ff6e74dd6c6b905543b7fa62c92: Status 404 returned error can't find the container with id fd734ec20c3d86d1b6b140e0990f95bb5e967ff6e74dd6c6b905543b7fa62c92 Jul 17 18:46:27.163166 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:27.163130 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" event={"ID":"1153df46-a624-48d4-bbd1-a3d2154479d3","Type":"ContainerStarted","Data":"fd734ec20c3d86d1b6b140e0990f95bb5e967ff6e74dd6c6b905543b7fa62c92"} Jul 17 18:46:29.184297 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:29.184253 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" event={"ID":"1153df46-a624-48d4-bbd1-a3d2154479d3","Type":"ContainerStarted","Data":"00b7b245918abd49510ad3dbad4a51ee38a4f4d39dfa15ea3ee6ac5fe32871f5"} Jul 17 18:46:29.202008 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:29.201942 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" podStartSLOduration=2.023510277 podStartE2EDuration="3.20192391s" totalImagesPullingTime="1.178413633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:29.200614156 +0000 UTC m=+806.104188032" watchObservedRunningTime="2026-07-17 18:46:29.20192391 +0000 UTC m=+806.105497786" Jul 17 18:46:35.013918 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.013882 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm"] Jul 17 18:46:35.022237 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.022200 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.025076 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.025041 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 17 18:46:35.026751 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.026716 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm"] Jul 17 18:46:35.074039 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.073995 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.074215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.074061 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.074215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.074111 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.074215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.074140 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.074215 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.074204 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.074350 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.074244 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bcmw\" (UniqueName: \"kubernetes.io/projected/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-kube-api-access-8bcmw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.174871 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.174834 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.174871 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.174874 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.175126 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.174896 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.175126 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.174928 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.175126 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.174965 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bcmw\" (UniqueName: \"kubernetes.io/projected/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-kube-api-access-8bcmw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.175126 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.175004 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.175332 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.175311 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.175391 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.175353 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.175442 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.175403 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.177561 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.177537 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.177720 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.177700 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.183830 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.183800 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bcmw\" (UniqueName: \"kubernetes.io/projected/be1afb96-0ad6-4ef5-abe2-3fc54bd54674-kube-api-access-8bcmw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vmphm\" (UID: \"be1afb96-0ad6-4ef5-abe2-3fc54bd54674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.334229 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.334129 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:35.482079 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:35.482051 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm"] Jul 17 18:46:35.484378 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:46:35.484341 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe1afb96_0ad6_4ef5_abe2_3fc54bd54674.slice/crio-2a0162da59c4b3b3c77fcd4218a4f2e5a0311ee10419aba56479f6671f3dd6a5 WatchSource:0}: Error finding container 2a0162da59c4b3b3c77fcd4218a4f2e5a0311ee10419aba56479f6671f3dd6a5: Status 404 returned error can't find the container with id 2a0162da59c4b3b3c77fcd4218a4f2e5a0311ee10419aba56479f6671f3dd6a5 Jul 17 18:46:36.222911 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:36.222878 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" event={"ID":"be1afb96-0ad6-4ef5-abe2-3fc54bd54674","Type":"ContainerStarted","Data":"7a55e087aa06f59ab1de2a264ac10f1f0fcdae4df7820008b25d1319a2ae6e9f"} Jul 17 18:46:36.223355 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:36.222932 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" event={"ID":"be1afb96-0ad6-4ef5-abe2-3fc54bd54674","Type":"ContainerStarted","Data":"2a0162da59c4b3b3c77fcd4218a4f2e5a0311ee10419aba56479f6671f3dd6a5"} Jul 17 18:46:36.240483 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:36.240404 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" podStartSLOduration=2.055020181 podStartE2EDuration="2.240387467s" totalImagesPullingTime="185.367286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:46:36.238562042 +0000 UTC m=+813.142135919" watchObservedRunningTime="2026-07-17 18:46:36.240387467 +0000 UTC m=+813.143961343" Jul 17 18:46:36.810373 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:36.810331 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:36.810570 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:36.810418 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:36.823316 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:36.823282 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:37.243915 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:37.243878 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jpknd" Jul 17 18:46:45.335302 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:45.335256 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:45.335788 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:45.335344 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:45.347909 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:45.347883 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:46:46.295850 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:46:46.295818 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vmphm" Jul 17 18:47:10.318260 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.318225 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj"] Jul 17 18:47:10.324077 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.324049 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.326827 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.326802 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 17 18:47:10.335315 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.335284 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj"] Jul 17 18:47:10.402787 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.402751 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.402787 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.402791 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.402991 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.402843 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.402991 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.402859 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.402991 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.402889 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgcrf\" (UniqueName: \"kubernetes.io/projected/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-kube-api-access-vgcrf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.402991 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.402944 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.503860 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.503825 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.504068 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.503866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.504068 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.503908 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgcrf\" (UniqueName: \"kubernetes.io/projected/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-kube-api-access-vgcrf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.504068 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.503944 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.504068 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.504038 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.504068 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.504063 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.504413 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.504391 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.504515 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.504431 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.504515 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.504506 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.506338 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.506321 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.506691 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.506675 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.514914 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.514888 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgcrf\" (UniqueName: \"kubernetes.io/projected/7e0b7e4b-1dcd-494d-84e8-84e5203783b0-kube-api-access-vgcrf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj\" (UID: \"7e0b7e4b-1dcd-494d-84e8-84e5203783b0\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.634507 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.634392 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:10.807082 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:10.807025 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj"] Jul 17 18:47:10.815958 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:47:10.815925 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e0b7e4b_1dcd_494d_84e8_84e5203783b0.slice/crio-9dcd4b480d40a8a6f230a18ba1fb7dc5c75ea53daeefba0e7a1dfbbe345d12fe WatchSource:0}: Error finding container 9dcd4b480d40a8a6f230a18ba1fb7dc5c75ea53daeefba0e7a1dfbbe345d12fe: Status 404 returned error can't find the container with id 9dcd4b480d40a8a6f230a18ba1fb7dc5c75ea53daeefba0e7a1dfbbe345d12fe Jul 17 18:47:11.422410 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:11.422373 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" event={"ID":"7e0b7e4b-1dcd-494d-84e8-84e5203783b0","Type":"ContainerStarted","Data":"0b019117235108e4d7f81aaf927ab97882455691b43ee0073dc80293e149e1c9"} Jul 17 18:47:11.422410 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:11.422416 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" event={"ID":"7e0b7e4b-1dcd-494d-84e8-84e5203783b0","Type":"ContainerStarted","Data":"9dcd4b480d40a8a6f230a18ba1fb7dc5c75ea53daeefba0e7a1dfbbe345d12fe"} Jul 17 18:47:11.445002 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:11.444936 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" podStartSLOduration=1.176600638 podStartE2EDuration="1.444912861s" totalImagesPullingTime="268.312223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:47:11.443299176 +0000 UTC m=+848.346873052" watchObservedRunningTime="2026-07-17 18:47:11.444912861 +0000 UTC m=+848.348486743" Jul 17 18:47:20.634775 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:20.634732 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:20.635282 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:20.634932 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:20.647821 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:20.647790 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:47:21.491402 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:47:21.491372 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj" Jul 17 18:48:09.784936 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:09.784897 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f88c8798d-7vq99"] Jul 17 18:48:09.788878 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:09.788852 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f88c8798d-7vq99" Jul 17 18:48:09.802995 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:09.802955 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f88c8798d-7vq99"] Jul 17 18:48:09.871890 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:09.871853 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6f21b3f-e568-40e0-a58e-cef553291f80-tls-cert\") pod \"authorino-6f88c8798d-7vq99\" (UID: \"b6f21b3f-e568-40e0-a58e-cef553291f80\") " pod="kuadrant-system/authorino-6f88c8798d-7vq99" Jul 17 18:48:09.871890 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:09.871900 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9d8r\" (UniqueName: \"kubernetes.io/projected/b6f21b3f-e568-40e0-a58e-cef553291f80-kube-api-access-f9d8r\") pod \"authorino-6f88c8798d-7vq99\" (UID: \"b6f21b3f-e568-40e0-a58e-cef553291f80\") " pod="kuadrant-system/authorino-6f88c8798d-7vq99" Jul 17 18:48:09.973223 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:09.973176 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6f21b3f-e568-40e0-a58e-cef553291f80-tls-cert\") pod \"authorino-6f88c8798d-7vq99\" (UID: \"b6f21b3f-e568-40e0-a58e-cef553291f80\") " pod="kuadrant-system/authorino-6f88c8798d-7vq99" Jul 17 18:48:09.973406 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:09.973236 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9d8r\" (UniqueName: \"kubernetes.io/projected/b6f21b3f-e568-40e0-a58e-cef553291f80-kube-api-access-f9d8r\") pod \"authorino-6f88c8798d-7vq99\" (UID: \"b6f21b3f-e568-40e0-a58e-cef553291f80\") " pod="kuadrant-system/authorino-6f88c8798d-7vq99" Jul 17 18:48:09.975908 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:09.975877 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6f21b3f-e568-40e0-a58e-cef553291f80-tls-cert\") pod \"authorino-6f88c8798d-7vq99\" (UID: \"b6f21b3f-e568-40e0-a58e-cef553291f80\") " pod="kuadrant-system/authorino-6f88c8798d-7vq99" Jul 17 18:48:09.983305 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:09.983264 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9d8r\" (UniqueName: \"kubernetes.io/projected/b6f21b3f-e568-40e0-a58e-cef553291f80-kube-api-access-f9d8r\") pod \"authorino-6f88c8798d-7vq99\" (UID: \"b6f21b3f-e568-40e0-a58e-cef553291f80\") " pod="kuadrant-system/authorino-6f88c8798d-7vq99" Jul 17 18:48:10.098317 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:10.098228 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f88c8798d-7vq99" Jul 17 18:48:10.269726 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:10.269696 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f88c8798d-7vq99"] Jul 17 18:48:10.269872 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:48:10.269770 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6f21b3f_e568_40e0_a58e_cef553291f80.slice/crio-b22092a559e2a4d7ac39ca0d2bb74375d3d4fe6a8b1e4ecbe4c5a3524067fa5a WatchSource:0}: Error finding container b22092a559e2a4d7ac39ca0d2bb74375d3d4fe6a8b1e4ecbe4c5a3524067fa5a: Status 404 returned error can't find the container with id b22092a559e2a4d7ac39ca0d2bb74375d3d4fe6a8b1e4ecbe4c5a3524067fa5a Jul 17 18:48:10.271220 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:10.271195 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 18:48:10.760051 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:10.760016 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f88c8798d-7vq99" event={"ID":"b6f21b3f-e568-40e0-a58e-cef553291f80","Type":"ContainerStarted","Data":"b22092a559e2a4d7ac39ca0d2bb74375d3d4fe6a8b1e4ecbe4c5a3524067fa5a"} Jul 17 18:48:11.770470 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:11.770402 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f88c8798d-7vq99" event={"ID":"b6f21b3f-e568-40e0-a58e-cef553291f80","Type":"ContainerStarted","Data":"2993deb40380e25a7c74fc196647da2d63db95f5865ed6d17b83abeb438e5a14"} Jul 17 18:48:11.814192 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:11.814114 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f88c8798d-7vq99" podStartSLOduration=2.348624706 podStartE2EDuration="2.814085427s" totalImagesPullingTime="465.460721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:48:11.809565825 +0000 UTC m=+908.713139702" watchObservedRunningTime="2026-07-17 18:48:11.814085427 +0000 UTC m=+908.717659305" Jul 17 18:48:11.888751 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:11.888706 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75dc8b6b65-6brv9"] Jul 17 18:48:11.889183 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:11.889154 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75dc8b6b65-6brv9" podUID="9abcde45-c20a-4b5e-a4ee-5c61b621ff36" containerName="authorino" containerID="cri-o://78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb" gracePeriod=30 Jul 17 18:48:12.176662 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.176632 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75dc8b6b65-6brv9" Jul 17 18:48:12.194571 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.194529 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbbc7\" (UniqueName: \"kubernetes.io/projected/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-kube-api-access-dbbc7\") pod \"9abcde45-c20a-4b5e-a4ee-5c61b621ff36\" (UID: \"9abcde45-c20a-4b5e-a4ee-5c61b621ff36\") " Jul 17 18:48:12.194770 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.194598 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-tls-cert\") pod \"9abcde45-c20a-4b5e-a4ee-5c61b621ff36\" (UID: \"9abcde45-c20a-4b5e-a4ee-5c61b621ff36\") " Jul 17 18:48:12.197042 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.197007 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-kube-api-access-dbbc7" (OuterVolumeSpecName: "kube-api-access-dbbc7") pod "9abcde45-c20a-4b5e-a4ee-5c61b621ff36" (UID: "9abcde45-c20a-4b5e-a4ee-5c61b621ff36"). InnerVolumeSpecName "kube-api-access-dbbc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:48:12.235641 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.229951 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9abcde45-c20a-4b5e-a4ee-5c61b621ff36" (UID: "9abcde45-c20a-4b5e-a4ee-5c61b621ff36"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 18:48:12.295672 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.295631 2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-tls-cert\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:48:12.295672 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.295666 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbbc7\" (UniqueName: \"kubernetes.io/projected/9abcde45-c20a-4b5e-a4ee-5c61b621ff36-kube-api-access-dbbc7\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:48:12.778519 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.778466 2563 generic.go:356] "Generic (PLEG): container finished" podID="9abcde45-c20a-4b5e-a4ee-5c61b621ff36" containerID="78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb" exitCode=0 Jul 17 18:48:12.778985 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.778516 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75dc8b6b65-6brv9" event={"ID":"9abcde45-c20a-4b5e-a4ee-5c61b621ff36","Type":"ContainerDied","Data":"78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb"} Jul 17 18:48:12.778985 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.778582 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75dc8b6b65-6brv9" event={"ID":"9abcde45-c20a-4b5e-a4ee-5c61b621ff36","Type":"ContainerDied","Data":"f06de4874a7117c31bb2d0d12ed69d4160a72720d8c1af4b98a747e2da597681"} Jul 17 18:48:12.778985 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.778606 2563 scope.go:117] "RemoveContainer" containerID="78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb" Jul 17 18:48:12.778985 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.778536 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75dc8b6b65-6brv9" Jul 17 18:48:12.789303 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.789280 2563 scope.go:117] "RemoveContainer" containerID="78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb" Jul 17 18:48:12.789661 ip-10-0-132-220 kubenswrapper[2563]: E0717 18:48:12.789640 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb\": container with ID starting with 78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb not found: ID does not exist" containerID="78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb" Jul 17 18:48:12.789751 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.789669 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb"} err="failed to get container status \"78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb\": rpc error: code = NotFound desc = could not find container \"78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb\": container with ID starting with 78f62cb01878e6a5e0c2093d437a85611a75b87f2e52d3037f058a5c20b71ebb not found: ID does not exist" Jul 17 18:48:12.811224 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.811184 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75dc8b6b65-6brv9"] Jul 17 18:48:12.817808 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:12.817777 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75dc8b6b65-6brv9"] Jul 17 18:48:13.625744 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:48:13.625709 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9abcde45-c20a-4b5e-a4ee-5c61b621ff36" path="/var/lib/kubelet/pods/9abcde45-c20a-4b5e-a4ee-5c61b621ff36/volumes" Jul 17 18:55:03.132114 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:03.132076 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5"] Jul 17 18:55:03.134524 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:03.132330 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" podUID="61879909-0d16-4f8f-80b8-154ac8d68b73" containerName="manager" containerID="cri-o://625aef2eb723afcb3976d93dc162f0977a5eab76ed1332f71211077b15a3263e" gracePeriod=10 Jul 17 18:55:03.498418 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:03.498393 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:55:03.593743 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:03.593704 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61879909-0d16-4f8f-80b8-154ac8d68b73-extensions-socket-volume\") pod \"61879909-0d16-4f8f-80b8-154ac8d68b73\" (UID: \"61879909-0d16-4f8f-80b8-154ac8d68b73\") " Jul 17 18:55:03.593924 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:03.593817 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx4tl\" (UniqueName: \"kubernetes.io/projected/61879909-0d16-4f8f-80b8-154ac8d68b73-kube-api-access-vx4tl\") pod \"61879909-0d16-4f8f-80b8-154ac8d68b73\" (UID: \"61879909-0d16-4f8f-80b8-154ac8d68b73\") " Jul 17 18:55:03.594085 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:03.594061 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61879909-0d16-4f8f-80b8-154ac8d68b73-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "61879909-0d16-4f8f-80b8-154ac8d68b73" (UID: "61879909-0d16-4f8f-80b8-154ac8d68b73"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 18:55:03.595994 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:03.595967 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61879909-0d16-4f8f-80b8-154ac8d68b73-kube-api-access-vx4tl" (OuterVolumeSpecName: "kube-api-access-vx4tl") pod "61879909-0d16-4f8f-80b8-154ac8d68b73" (UID: "61879909-0d16-4f8f-80b8-154ac8d68b73"). InnerVolumeSpecName "kube-api-access-vx4tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 18:55:03.677826 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:03.677741 2563 scope.go:117] "RemoveContainer" containerID="625aef2eb723afcb3976d93dc162f0977a5eab76ed1332f71211077b15a3263e" Jul 17 18:55:03.695198 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:03.695173 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx4tl\" (UniqueName: \"kubernetes.io/projected/61879909-0d16-4f8f-80b8-154ac8d68b73-kube-api-access-vx4tl\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:55:03.695198 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:03.695197 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61879909-0d16-4f8f-80b8-154ac8d68b73-extensions-socket-volume\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 18:55:04.060151 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:04.060065 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" Jul 17 18:55:04.060151 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:04.060081 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" event={"ID":"61879909-0d16-4f8f-80b8-154ac8d68b73","Type":"ContainerDied","Data":"625aef2eb723afcb3976d93dc162f0977a5eab76ed1332f71211077b15a3263e"} Jul 17 18:55:04.060151 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:04.060143 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5" event={"ID":"61879909-0d16-4f8f-80b8-154ac8d68b73","Type":"ContainerDied","Data":"7fce5ed55eb79bcc08fea0b2e4529b0d971bf5c7c5a07e825edf2a0a5c59323d"} Jul 17 18:55:04.092037 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:04.091999 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5"] Jul 17 18:55:04.098234 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:04.098205 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvvw5"] Jul 17 18:55:05.623020 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:55:05.622981 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61879909-0d16-4f8f-80b8-154ac8d68b73" path="/var/lib/kubelet/pods/61879909-0d16-4f8f-80b8-154ac8d68b73/volumes" Jul 17 18:56:04.270392 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.270354 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6"] Jul 17 18:56:04.270912 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.270810 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abcde45-c20a-4b5e-a4ee-5c61b621ff36" containerName="authorino" Jul 17 18:56:04.270912 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.270823 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abcde45-c20a-4b5e-a4ee-5c61b621ff36" containerName="authorino" Jul 17 18:56:04.270912 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.270837 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61879909-0d16-4f8f-80b8-154ac8d68b73" containerName="manager" Jul 17 18:56:04.270912 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.270843 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="61879909-0d16-4f8f-80b8-154ac8d68b73" containerName="manager" Jul 17 18:56:04.271043 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.270917 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="61879909-0d16-4f8f-80b8-154ac8d68b73" containerName="manager" Jul 17 18:56:04.271043 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.270931 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9abcde45-c20a-4b5e-a4ee-5c61b621ff36" containerName="authorino" Jul 17 18:56:04.274023 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.274002 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" Jul 17 18:56:04.276851 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.276823 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jb9j9\"" Jul 17 18:56:04.292216 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.287070 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6"] Jul 17 18:56:04.371533 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.371484 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37c99017-dcf8-4054-bc3e-6de71d9d5141-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fl9s6\" (UID: \"37c99017-dcf8-4054-bc3e-6de71d9d5141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" Jul 17 18:56:04.371735 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.371574 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwb4v\" (UniqueName: \"kubernetes.io/projected/37c99017-dcf8-4054-bc3e-6de71d9d5141-kube-api-access-dwb4v\") pod \"kuadrant-operator-controller-manager-86878c769f-fl9s6\" (UID: \"37c99017-dcf8-4054-bc3e-6de71d9d5141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" Jul 17 18:56:04.472441 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.472403 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwb4v\" (UniqueName: \"kubernetes.io/projected/37c99017-dcf8-4054-bc3e-6de71d9d5141-kube-api-access-dwb4v\") pod \"kuadrant-operator-controller-manager-86878c769f-fl9s6\" (UID: \"37c99017-dcf8-4054-bc3e-6de71d9d5141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" Jul 17 18:56:04.472627 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.472582 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37c99017-dcf8-4054-bc3e-6de71d9d5141-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fl9s6\" (UID: \"37c99017-dcf8-4054-bc3e-6de71d9d5141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" Jul 17 18:56:04.472997 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.472971 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/37c99017-dcf8-4054-bc3e-6de71d9d5141-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fl9s6\" (UID: \"37c99017-dcf8-4054-bc3e-6de71d9d5141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" Jul 17 18:56:04.484041 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.483998 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwb4v\" (UniqueName: \"kubernetes.io/projected/37c99017-dcf8-4054-bc3e-6de71d9d5141-kube-api-access-dwb4v\") pod \"kuadrant-operator-controller-manager-86878c769f-fl9s6\" (UID: \"37c99017-dcf8-4054-bc3e-6de71d9d5141\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" Jul 17 18:56:04.594020 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.593918 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" Jul 17 18:56:04.761464 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:04.761409 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6"] Jul 17 18:56:04.765325 ip-10-0-132-220 kubenswrapper[2563]: W0717 18:56:04.765286 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c99017_dcf8_4054_bc3e_6de71d9d5141.slice/crio-5825111e1049a86958baabc94673951c259d196588eb0fe97545e2954067c0f5 WatchSource:0}: Error finding container 5825111e1049a86958baabc94673951c259d196588eb0fe97545e2954067c0f5: Status 404 returned error can't find the container with id 5825111e1049a86958baabc94673951c259d196588eb0fe97545e2954067c0f5 Jul 17 18:56:05.402539 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:05.402502 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" event={"ID":"37c99017-dcf8-4054-bc3e-6de71d9d5141","Type":"ContainerStarted","Data":"172214752f3c1da1e4a5537f0b8c218adc8da8cabb85ce747ad23363c322ecea"} Jul 17 18:56:05.403046 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:05.402562 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" event={"ID":"37c99017-dcf8-4054-bc3e-6de71d9d5141","Type":"ContainerStarted","Data":"5825111e1049a86958baabc94673951c259d196588eb0fe97545e2954067c0f5"} Jul 17 18:56:05.403046 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:05.402721 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" Jul 17 18:56:05.430293 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:05.430235 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" podStartSLOduration=1.430216155 podStartE2EDuration="1.430216155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 18:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 18:56:05.426812864 +0000 UTC m=+1382.330386740" watchObservedRunningTime="2026-07-17 18:56:05.430216155 +0000 UTC m=+1382.333790030" Jul 17 18:56:16.411307 ip-10-0-132-220 kubenswrapper[2563]: I0717 18:56:16.411270 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fl9s6" Jul 17 19:03:44.976535 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:44.976493 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n"] Jul 17 19:03:44.980221 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:44.980198 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" Jul 17 19:03:44.982728 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:44.982700 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:03:44.982870 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:44.982775 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\"" Jul 17 19:03:44.990177 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:44.990137 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n"] Jul 17 19:03:45.132567 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:45.132521 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c207f444-1d07-48f0-a72f-8985c673bc07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n\" (UID: \"c207f444-1d07-48f0-a72f-8985c673bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" Jul 17 19:03:45.233661 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:45.233569 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c207f444-1d07-48f0-a72f-8985c673bc07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n\" (UID: \"c207f444-1d07-48f0-a72f-8985c673bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" Jul 17 19:03:45.234223 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:45.234202 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c207f444-1d07-48f0-a72f-8985c673bc07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n\" (UID: \"c207f444-1d07-48f0-a72f-8985c673bc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" Jul 17 19:03:45.293600 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:45.293553 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" Jul 17 19:03:45.442389 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:45.442364 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n"] Jul 17 19:03:45.444621 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:03:45.444588 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc207f444_1d07_48f0_a72f_8985c673bc07.slice/crio-9a83fd709d455d160d9c098ef534b9936cb138ad47e861cb795446fbacbbe6db WatchSource:0}: Error finding container 9a83fd709d455d160d9c098ef534b9936cb138ad47e861cb795446fbacbbe6db: Status 404 returned error can't find the container with id 9a83fd709d455d160d9c098ef534b9936cb138ad47e861cb795446fbacbbe6db Jul 17 19:03:45.446563 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:45.446546 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 19:03:46.012604 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:46.012567 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" event={"ID":"c207f444-1d07-48f0-a72f-8985c673bc07","Type":"ContainerStarted","Data":"9a83fd709d455d160d9c098ef534b9936cb138ad47e861cb795446fbacbbe6db"} Jul 17 19:03:48.026825 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:48.026781 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" event={"ID":"c207f444-1d07-48f0-a72f-8985c673bc07","Type":"ContainerStarted","Data":"61ec0c7c0c544ca810485026a0073d069a49111feed618ad61a508a6434ac77c"} Jul 17 19:03:48.044568 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:48.044488 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" podStartSLOduration=2.09381167 podStartE2EDuration="4.044466876s" totalImagesPullingTime="1.950655206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:03:48.044099878 +0000 UTC m=+1844.947673756" watchObservedRunningTime="2026-07-17 19:03:48.044466876 +0000 UTC m=+1844.948040745" Jul 17 19:03:49.034297 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:49.034204 2563 generic.go:356] "Generic (PLEG): container finished" podID="c207f444-1d07-48f0-a72f-8985c673bc07" containerID="61ec0c7c0c544ca810485026a0073d069a49111feed618ad61a508a6434ac77c" exitCode=7 Jul 17 19:03:49.034688 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:49.034297 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" event={"ID":"c207f444-1d07-48f0-a72f-8985c673bc07","Type":"ContainerDied","Data":"61ec0c7c0c544ca810485026a0073d069a49111feed618ad61a508a6434ac77c"} Jul 17 19:03:49.034688 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:49.034540 2563 scope.go:117] "RemoveContainer" containerID="61ec0c7c0c544ca810485026a0073d069a49111feed618ad61a508a6434ac77c" Jul 17 19:03:50.042647 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:50.042614 2563 generic.go:356] "Generic (PLEG): container finished" podID="c207f444-1d07-48f0-a72f-8985c673bc07" containerID="3f06db2a7d85fdf4f8a47bbd10d0a7a860b1637774b10ab6aafcc7a883b268d2" exitCode=7 Jul 17 19:03:50.043152 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:50.042699 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" event={"ID":"c207f444-1d07-48f0-a72f-8985c673bc07","Type":"ContainerDied","Data":"3f06db2a7d85fdf4f8a47bbd10d0a7a860b1637774b10ab6aafcc7a883b268d2"} Jul 17 19:03:50.043152 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:50.042765 2563 scope.go:117] "RemoveContainer" containerID="61ec0c7c0c544ca810485026a0073d069a49111feed618ad61a508a6434ac77c" Jul 17 19:03:50.043152 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:50.042978 2563 scope.go:117] "RemoveContainer" containerID="3f06db2a7d85fdf4f8a47bbd10d0a7a860b1637774b10ab6aafcc7a883b268d2" Jul 17 19:03:50.043318 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:03:50.043271 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n_odh-ai-gateway-infra(c207f444-1d07-48f0-a72f-8985c673bc07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" Jul 17 19:03:51.050293 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:03:51.050267 2563 scope.go:117] "RemoveContainer" containerID="3f06db2a7d85fdf4f8a47bbd10d0a7a860b1637774b10ab6aafcc7a883b268d2" Jul 17 19:03:51.050794 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:03:51.050509 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n_odh-ai-gateway-infra(c207f444-1d07-48f0-a72f-8985c673bc07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" Jul 17 19:04:05.616898 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:05.616858 2563 scope.go:117] "RemoveContainer" containerID="3f06db2a7d85fdf4f8a47bbd10d0a7a860b1637774b10ab6aafcc7a883b268d2" Jul 17 19:04:06.131537 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:06.131498 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" event={"ID":"c207f444-1d07-48f0-a72f-8985c673bc07","Type":"ContainerStarted","Data":"34918366bda2b66dcbc91bcc6600ebf404253075baeefdcc923babfe61abe1ca"} Jul 17 19:04:07.138783 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:07.138748 2563 generic.go:356] "Generic (PLEG): container finished" podID="c207f444-1d07-48f0-a72f-8985c673bc07" containerID="34918366bda2b66dcbc91bcc6600ebf404253075baeefdcc923babfe61abe1ca" exitCode=7 Jul 17 19:04:07.139219 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:07.138821 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" event={"ID":"c207f444-1d07-48f0-a72f-8985c673bc07","Type":"ContainerDied","Data":"34918366bda2b66dcbc91bcc6600ebf404253075baeefdcc923babfe61abe1ca"} Jul 17 19:04:07.139219 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:07.138883 2563 scope.go:117] "RemoveContainer" containerID="3f06db2a7d85fdf4f8a47bbd10d0a7a860b1637774b10ab6aafcc7a883b268d2" Jul 17 19:04:07.139219 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:07.139214 2563 scope.go:117] "RemoveContainer" containerID="34918366bda2b66dcbc91bcc6600ebf404253075baeefdcc923babfe61abe1ca" Jul 17 19:04:07.139494 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:04:07.139466 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n_odh-ai-gateway-infra(c207f444-1d07-48f0-a72f-8985c673bc07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" Jul 17 19:04:07.166873 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:07.166838 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n"] Jul 17 19:04:08.301111 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:08.301075 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" Jul 17 19:04:08.351693 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:08.351654 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c207f444-1d07-48f0-a72f-8985c673bc07-maas-api-service-ca\") pod \"c207f444-1d07-48f0-a72f-8985c673bc07\" (UID: \"c207f444-1d07-48f0-a72f-8985c673bc07\") " Jul 17 19:04:08.352025 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:08.351998 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c207f444-1d07-48f0-a72f-8985c673bc07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c207f444-1d07-48f0-a72f-8985c673bc07" (UID: "c207f444-1d07-48f0-a72f-8985c673bc07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:04:08.452796 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:08.452757 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c207f444-1d07-48f0-a72f-8985c673bc07-maas-api-service-ca\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:04:09.155063 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:09.155023 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" event={"ID":"c207f444-1d07-48f0-a72f-8985c673bc07","Type":"ContainerDied","Data":"9a83fd709d455d160d9c098ef534b9936cb138ad47e861cb795446fbacbbe6db"} Jul 17 19:04:09.155063 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:09.155062 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n" Jul 17 19:04:09.155309 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:09.155103 2563 scope.go:117] "RemoveContainer" containerID="34918366bda2b66dcbc91bcc6600ebf404253075baeefdcc923babfe61abe1ca" Jul 17 19:04:09.178726 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:09.178688 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n"] Jul 17 19:04:09.186083 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:09.186047 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-jgh4n"] Jul 17 19:04:09.624492 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:04:09.624387 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" path="/var/lib/kubelet/pods/c207f444-1d07-48f0-a72f-8985c673bc07/volumes" Jul 17 19:05:35.071021 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.070980 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr"] Jul 17 19:05:35.071552 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.071422 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" containerName="revoke-keys" Jul 17 19:05:35.071552 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.071435 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" containerName="revoke-keys" Jul 17 19:05:35.071552 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.071457 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" containerName="revoke-keys" Jul 17 19:05:35.071552 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.071463 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" containerName="revoke-keys" Jul 17 19:05:35.071552 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.071476 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" containerName="revoke-keys" Jul 17 19:05:35.071552 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.071482 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" containerName="revoke-keys" Jul 17 19:05:35.071552 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.071541 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" containerName="revoke-keys" Jul 17 19:05:35.071552 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.071550 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" containerName="revoke-keys" Jul 17 19:05:35.071552 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.071557 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c207f444-1d07-48f0-a72f-8985c673bc07" containerName="revoke-keys" Jul 17 19:05:35.074796 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.074774 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" Jul 17 19:05:35.077531 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.077487 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:05:35.077650 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.077634 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\"" Jul 17 19:05:35.082633 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.082600 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr"] Jul 17 19:05:35.181080 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.181034 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c02d1c-9535-473c-b1e7-711e77f18c2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr\" (UID: \"d5c02d1c-9535-473c-b1e7-711e77f18c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" Jul 17 19:05:35.282321 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.282278 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c02d1c-9535-473c-b1e7-711e77f18c2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr\" (UID: \"d5c02d1c-9535-473c-b1e7-711e77f18c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" Jul 17 19:05:35.282972 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.282951 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c02d1c-9535-473c-b1e7-711e77f18c2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr\" (UID: \"d5c02d1c-9535-473c-b1e7-711e77f18c2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" Jul 17 19:05:35.387846 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.387736 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" Jul 17 19:05:35.536877 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.536850 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr"] Jul 17 19:05:35.539661 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:05:35.539627 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c02d1c_9535_473c_b1e7_711e77f18c2e.slice/crio-bb7b5dccbd44e5281de3f33a618c44dd3a854c15ef53d45de66172810f8e9c50 WatchSource:0}: Error finding container bb7b5dccbd44e5281de3f33a618c44dd3a854c15ef53d45de66172810f8e9c50: Status 404 returned error can't find the container with id bb7b5dccbd44e5281de3f33a618c44dd3a854c15ef53d45de66172810f8e9c50 Jul 17 19:05:35.639101 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:35.639009 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" event={"ID":"d5c02d1c-9535-473c-b1e7-711e77f18c2e","Type":"ContainerStarted","Data":"bb7b5dccbd44e5281de3f33a618c44dd3a854c15ef53d45de66172810f8e9c50"} Jul 17 19:05:36.647880 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:36.647774 2563 generic.go:356] "Generic (PLEG): container finished" podID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerID="6a995a5e727fcad4509eab783313ba4dfb12eebf319101eb5a67ec05e95cf916" exitCode=7 Jul 17 19:05:36.648341 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:36.647877 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" event={"ID":"d5c02d1c-9535-473c-b1e7-711e77f18c2e","Type":"ContainerDied","Data":"6a995a5e727fcad4509eab783313ba4dfb12eebf319101eb5a67ec05e95cf916"} Jul 17 19:05:36.648341 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:36.648161 2563 scope.go:117] "RemoveContainer" containerID="6a995a5e727fcad4509eab783313ba4dfb12eebf319101eb5a67ec05e95cf916" Jul 17 19:05:37.655737 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:37.655707 2563 generic.go:356] "Generic (PLEG): container finished" podID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerID="d16a2065f30592862cf5e502575a75ee571bcdc49bb4cb5275cd211eba0b1f88" exitCode=7 Jul 17 19:05:37.656199 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:37.655800 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" event={"ID":"d5c02d1c-9535-473c-b1e7-711e77f18c2e","Type":"ContainerDied","Data":"d16a2065f30592862cf5e502575a75ee571bcdc49bb4cb5275cd211eba0b1f88"} Jul 17 19:05:37.656199 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:37.655859 2563 scope.go:117] "RemoveContainer" containerID="6a995a5e727fcad4509eab783313ba4dfb12eebf319101eb5a67ec05e95cf916" Jul 17 19:05:37.656199 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:37.655970 2563 scope.go:117] "RemoveContainer" containerID="d16a2065f30592862cf5e502575a75ee571bcdc49bb4cb5275cd211eba0b1f88" Jul 17 19:05:37.656371 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:05:37.656230 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr_odh-ai-gateway-infra(d5c02d1c-9535-473c-b1e7-711e77f18c2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" Jul 17 19:05:38.663607 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:38.663574 2563 scope.go:117] "RemoveContainer" containerID="d16a2065f30592862cf5e502575a75ee571bcdc49bb4cb5275cd211eba0b1f88" Jul 17 19:05:38.663977 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:05:38.663808 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr_odh-ai-gateway-infra(d5c02d1c-9535-473c-b1e7-711e77f18c2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" Jul 17 19:05:49.616624 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:49.616590 2563 scope.go:117] "RemoveContainer" containerID="d16a2065f30592862cf5e502575a75ee571bcdc49bb4cb5275cd211eba0b1f88" Jul 17 19:05:50.731758 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:50.731724 2563 generic.go:356] "Generic (PLEG): container finished" podID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerID="94b53a4dd8d5a7d54e89127b1ff5f4e794505625d33d56ed2518a092d6d7bf76" exitCode=7 Jul 17 19:05:50.732304 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:50.731802 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" event={"ID":"d5c02d1c-9535-473c-b1e7-711e77f18c2e","Type":"ContainerDied","Data":"94b53a4dd8d5a7d54e89127b1ff5f4e794505625d33d56ed2518a092d6d7bf76"} Jul 17 19:05:50.732304 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:50.731867 2563 scope.go:117] "RemoveContainer" containerID="d16a2065f30592862cf5e502575a75ee571bcdc49bb4cb5275cd211eba0b1f88" Jul 17 19:05:50.732304 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:50.732239 2563 scope.go:117] "RemoveContainer" containerID="94b53a4dd8d5a7d54e89127b1ff5f4e794505625d33d56ed2518a092d6d7bf76" Jul 17 19:05:50.732508 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:05:50.732487 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr_odh-ai-gateway-infra(d5c02d1c-9535-473c-b1e7-711e77f18c2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" Jul 17 19:05:51.761620 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:51.761582 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr"] Jul 17 19:05:51.919928 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:51.919889 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" Jul 17 19:05:52.040235 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:52.040120 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c02d1c-9535-473c-b1e7-711e77f18c2e-maas-api-service-ca\") pod \"d5c02d1c-9535-473c-b1e7-711e77f18c2e\" (UID: \"d5c02d1c-9535-473c-b1e7-711e77f18c2e\") " Jul 17 19:05:52.040497 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:52.040471 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c02d1c-9535-473c-b1e7-711e77f18c2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5c02d1c-9535-473c-b1e7-711e77f18c2e" (UID: "d5c02d1c-9535-473c-b1e7-711e77f18c2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:05:52.141680 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:52.141644 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c02d1c-9535-473c-b1e7-711e77f18c2e-maas-api-service-ca\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:05:52.747276 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:52.747239 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" event={"ID":"d5c02d1c-9535-473c-b1e7-711e77f18c2e","Type":"ContainerDied","Data":"bb7b5dccbd44e5281de3f33a618c44dd3a854c15ef53d45de66172810f8e9c50"} Jul 17 19:05:52.747510 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:52.747280 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr" Jul 17 19:05:52.747510 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:52.747305 2563 scope.go:117] "RemoveContainer" containerID="94b53a4dd8d5a7d54e89127b1ff5f4e794505625d33d56ed2518a092d6d7bf76" Jul 17 19:05:52.769038 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:52.768998 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr"] Jul 17 19:05:52.775226 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:52.775177 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-vnbrr"] Jul 17 19:05:53.622991 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:05:53.622907 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" path="/var/lib/kubelet/pods/d5c02d1c-9535-473c-b1e7-711e77f18c2e/volumes" Jul 17 19:19:55.773364 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.773334 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg"] Jul 17 19:19:55.773802 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.773789 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerName="revoke-keys" Jul 17 19:19:55.773802 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.773801 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerName="revoke-keys" Jul 17 19:19:55.773878 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.773812 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerName="revoke-keys" Jul 17 19:19:55.773878 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.773820 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerName="revoke-keys" Jul 17 19:19:55.773940 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.773892 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerName="revoke-keys" Jul 17 19:19:55.773940 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.773900 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerName="revoke-keys" Jul 17 19:19:55.773940 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.773908 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerName="revoke-keys" Jul 17 19:19:55.777178 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.777151 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" Jul 17 19:19:55.780067 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.780049 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 19:19:55.780178 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.780064 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkd9v\"" Jul 17 19:19:55.785550 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.785524 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg"] Jul 17 19:19:55.826996 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.826958 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd767773-8ee7-465d-b964-c7842db55e7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg\" (UID: \"dd767773-8ee7-465d-b964-c7842db55e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" Jul 17 19:19:55.928215 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.928179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd767773-8ee7-465d-b964-c7842db55e7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg\" (UID: \"dd767773-8ee7-465d-b964-c7842db55e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" Jul 17 19:19:55.928892 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:55.928863 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd767773-8ee7-465d-b964-c7842db55e7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg\" (UID: \"dd767773-8ee7-465d-b964-c7842db55e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" Jul 17 19:19:56.089485 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:56.089362 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" Jul 17 19:19:56.442036 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:56.442012 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg"] Jul 17 19:19:56.445181 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:19:56.445135 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd767773_8ee7_465d_b964_c7842db55e7f.slice/crio-61c4ed80b2e4e8914e2ef076c871504b923a7b408727fba88e187c7d3ea0e43f WatchSource:0}: Error finding container 61c4ed80b2e4e8914e2ef076c871504b923a7b408727fba88e187c7d3ea0e43f: Status 404 returned error can't find the container with id 61c4ed80b2e4e8914e2ef076c871504b923a7b408727fba88e187c7d3ea0e43f Jul 17 19:19:57.324048 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:57.324011 2563 generic.go:356] "Generic (PLEG): container finished" podID="dd767773-8ee7-465d-b964-c7842db55e7f" containerID="5519040bdfaa24c6c0cc17e23d52c5c71881cf7c9997b726462323ecebe56d85" exitCode=6 Jul 17 19:19:57.324540 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:57.324109 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" event={"ID":"dd767773-8ee7-465d-b964-c7842db55e7f","Type":"ContainerDied","Data":"5519040bdfaa24c6c0cc17e23d52c5c71881cf7c9997b726462323ecebe56d85"} Jul 17 19:19:57.324540 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:57.324171 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" event={"ID":"dd767773-8ee7-465d-b964-c7842db55e7f","Type":"ContainerStarted","Data":"61c4ed80b2e4e8914e2ef076c871504b923a7b408727fba88e187c7d3ea0e43f"} Jul 17 19:19:57.324540 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:57.324282 2563 scope.go:117] "RemoveContainer" containerID="5519040bdfaa24c6c0cc17e23d52c5c71881cf7c9997b726462323ecebe56d85" Jul 17 19:19:58.331481 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:58.331360 2563 generic.go:356] "Generic (PLEG): container finished" podID="dd767773-8ee7-465d-b964-c7842db55e7f" containerID="d866d14d80f55e27db723863a2613e9914825310b1c628d2d908fd6822f3d95a" exitCode=6 Jul 17 19:19:58.331886 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:58.331478 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" event={"ID":"dd767773-8ee7-465d-b964-c7842db55e7f","Type":"ContainerDied","Data":"d866d14d80f55e27db723863a2613e9914825310b1c628d2d908fd6822f3d95a"} Jul 17 19:19:58.331886 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:58.331583 2563 scope.go:117] "RemoveContainer" containerID="5519040bdfaa24c6c0cc17e23d52c5c71881cf7c9997b726462323ecebe56d85" Jul 17 19:19:58.331886 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:58.331632 2563 scope.go:117] "RemoveContainer" containerID="d866d14d80f55e27db723863a2613e9914825310b1c628d2d908fd6822f3d95a" Jul 17 19:19:58.332004 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:19:58.331877 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg_odh-ai-gateway-infra(dd767773-8ee7-465d-b964-c7842db55e7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" Jul 17 19:19:59.338345 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:19:59.338308 2563 scope.go:117] "RemoveContainer" containerID="d866d14d80f55e27db723863a2613e9914825310b1c628d2d908fd6822f3d95a" Jul 17 19:19:59.338748 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:19:59.338543 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg_odh-ai-gateway-infra(dd767773-8ee7-465d-b964-c7842db55e7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" Jul 17 19:20:01.122540 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:01.122507 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk"] Jul 17 19:20:01.122995 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:01.122972 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerName="revoke-keys" Jul 17 19:20:01.122995 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:01.122990 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c02d1c-9535-473c-b1e7-711e77f18c2e" containerName="revoke-keys" Jul 17 19:20:01.127412 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:01.127395 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" Jul 17 19:20:01.136038 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:01.136014 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk"] Jul 17 19:20:01.177464 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:01.177414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2871553b-a031-4615-bca8-747cc11cd5c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk\" (UID: \"2871553b-a031-4615-bca8-747cc11cd5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" Jul 17 19:20:01.278664 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:01.278637 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2871553b-a031-4615-bca8-747cc11cd5c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk\" (UID: \"2871553b-a031-4615-bca8-747cc11cd5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" Jul 17 19:20:01.279228 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:01.279209 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2871553b-a031-4615-bca8-747cc11cd5c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk\" (UID: \"2871553b-a031-4615-bca8-747cc11cd5c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" Jul 17 19:20:01.438493 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:01.438432 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" Jul 17 19:20:01.790023 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:01.789995 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk"] Jul 17 19:20:01.792752 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:20:01.792722 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2871553b_a031_4615_bca8_747cc11cd5c6.slice/crio-d2058fe1f474e7ce361674dbdb8cd54e3e867d66c2d113ca5fe3cac3870aa52e WatchSource:0}: Error finding container d2058fe1f474e7ce361674dbdb8cd54e3e867d66c2d113ca5fe3cac3870aa52e: Status 404 returned error can't find the container with id d2058fe1f474e7ce361674dbdb8cd54e3e867d66c2d113ca5fe3cac3870aa52e Jul 17 19:20:02.356761 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:02.356676 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" event={"ID":"2871553b-a031-4615-bca8-747cc11cd5c6","Type":"ContainerStarted","Data":"c952b1fb6e21801947cf1cf06fde162c4c1d845f66d5852898913a6d1d117d8a"} Jul 17 19:20:02.356761 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:02.356732 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" event={"ID":"2871553b-a031-4615-bca8-747cc11cd5c6","Type":"ContainerStarted","Data":"d2058fe1f474e7ce361674dbdb8cd54e3e867d66c2d113ca5fe3cac3870aa52e"} Jul 17 19:20:02.381759 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:02.381706 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" podStartSLOduration=1.381689732 podStartE2EDuration="1.381689732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:20:02.38026908 +0000 UTC m=+2819.283842958" watchObservedRunningTime="2026-07-17 19:20:02.381689732 +0000 UTC m=+2819.285263607" Jul 17 19:20:03.362996 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:03.362954 2563 generic.go:356] "Generic (PLEG): container finished" podID="2871553b-a031-4615-bca8-747cc11cd5c6" containerID="c952b1fb6e21801947cf1cf06fde162c4c1d845f66d5852898913a6d1d117d8a" exitCode=6 Jul 17 19:20:03.363392 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:03.363043 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" event={"ID":"2871553b-a031-4615-bca8-747cc11cd5c6","Type":"ContainerDied","Data":"c952b1fb6e21801947cf1cf06fde162c4c1d845f66d5852898913a6d1d117d8a"} Jul 17 19:20:03.363392 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:03.363264 2563 scope.go:117] "RemoveContainer" containerID="c952b1fb6e21801947cf1cf06fde162c4c1d845f66d5852898913a6d1d117d8a" Jul 17 19:20:04.369750 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:04.369660 2563 generic.go:356] "Generic (PLEG): container finished" podID="2871553b-a031-4615-bca8-747cc11cd5c6" containerID="d717b7df167903fe440573b6478eaadfc3ba6bb42f4a87d8339a6a616d3af4b1" exitCode=6 Jul 17 19:20:04.370140 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:04.369748 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" event={"ID":"2871553b-a031-4615-bca8-747cc11cd5c6","Type":"ContainerDied","Data":"d717b7df167903fe440573b6478eaadfc3ba6bb42f4a87d8339a6a616d3af4b1"} Jul 17 19:20:04.370140 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:04.369818 2563 scope.go:117] "RemoveContainer" containerID="c952b1fb6e21801947cf1cf06fde162c4c1d845f66d5852898913a6d1d117d8a" Jul 17 19:20:04.370140 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:04.370027 2563 scope.go:117] "RemoveContainer" containerID="d717b7df167903fe440573b6478eaadfc3ba6bb42f4a87d8339a6a616d3af4b1" Jul 17 19:20:04.370265 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:20:04.370244 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk_odh-ai-gateway-infra(2871553b-a031-4615-bca8-747cc11cd5c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" Jul 17 19:20:05.376745 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:05.376719 2563 scope.go:117] "RemoveContainer" containerID="d717b7df167903fe440573b6478eaadfc3ba6bb42f4a87d8339a6a616d3af4b1" Jul 17 19:20:05.377146 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:20:05.376941 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk_odh-ai-gateway-infra(2871553b-a031-4615-bca8-747cc11cd5c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" Jul 17 19:20:11.617109 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:11.617072 2563 scope.go:117] "RemoveContainer" containerID="d866d14d80f55e27db723863a2613e9914825310b1c628d2d908fd6822f3d95a" Jul 17 19:20:12.416426 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:12.416396 2563 generic.go:356] "Generic (PLEG): container finished" podID="dd767773-8ee7-465d-b964-c7842db55e7f" containerID="bf6637d1ea3f2bf120ec4765246922d56b06cf47f30440d389fcb90d8a4c6d92" exitCode=6 Jul 17 19:20:12.416586 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:12.416501 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" event={"ID":"dd767773-8ee7-465d-b964-c7842db55e7f","Type":"ContainerDied","Data":"bf6637d1ea3f2bf120ec4765246922d56b06cf47f30440d389fcb90d8a4c6d92"} Jul 17 19:20:12.416586 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:12.416577 2563 scope.go:117] "RemoveContainer" containerID="d866d14d80f55e27db723863a2613e9914825310b1c628d2d908fd6822f3d95a" Jul 17 19:20:12.416924 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:12.416904 2563 scope.go:117] "RemoveContainer" containerID="bf6637d1ea3f2bf120ec4765246922d56b06cf47f30440d389fcb90d8a4c6d92" Jul 17 19:20:12.417118 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:20:12.417100 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg_odh-ai-gateway-infra(dd767773-8ee7-465d-b964-c7842db55e7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" Jul 17 19:20:13.453428 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:13.453395 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg"] Jul 17 19:20:13.605473 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:13.605432 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" Jul 17 19:20:13.684310 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:13.684282 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd767773-8ee7-465d-b964-c7842db55e7f-maas-api-service-ca\") pod \"dd767773-8ee7-465d-b964-c7842db55e7f\" (UID: \"dd767773-8ee7-465d-b964-c7842db55e7f\") " Jul 17 19:20:13.684641 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:13.684617 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd767773-8ee7-465d-b964-c7842db55e7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd767773-8ee7-465d-b964-c7842db55e7f" (UID: "dd767773-8ee7-465d-b964-c7842db55e7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:20:13.785236 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:13.785153 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd767773-8ee7-465d-b964-c7842db55e7f-maas-api-service-ca\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:20:14.429976 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:14.429943 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" Jul 17 19:20:14.429976 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:14.429966 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg" event={"ID":"dd767773-8ee7-465d-b964-c7842db55e7f","Type":"ContainerDied","Data":"61c4ed80b2e4e8914e2ef076c871504b923a7b408727fba88e187c7d3ea0e43f"} Jul 17 19:20:14.430224 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:14.430056 2563 scope.go:117] "RemoveContainer" containerID="bf6637d1ea3f2bf120ec4765246922d56b06cf47f30440d389fcb90d8a4c6d92" Jul 17 19:20:14.451672 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:14.451646 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg"] Jul 17 19:20:14.457854 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:14.457832 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7dc4fd53-p48lg"] Jul 17 19:20:15.623182 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:15.623146 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" path="/var/lib/kubelet/pods/dd767773-8ee7-465d-b964-c7842db55e7f/volumes" Jul 17 19:20:19.617336 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:19.617307 2563 scope.go:117] "RemoveContainer" containerID="d717b7df167903fe440573b6478eaadfc3ba6bb42f4a87d8339a6a616d3af4b1" Jul 17 19:20:20.471549 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:20.471509 2563 generic.go:356] "Generic (PLEG): container finished" podID="2871553b-a031-4615-bca8-747cc11cd5c6" containerID="f9432ff197188d5aa2f26c3bf4220365e3581430ce71142f1f6887801665c337" exitCode=6 Jul 17 19:20:20.471764 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:20.471578 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" event={"ID":"2871553b-a031-4615-bca8-747cc11cd5c6","Type":"ContainerDied","Data":"f9432ff197188d5aa2f26c3bf4220365e3581430ce71142f1f6887801665c337"} Jul 17 19:20:20.471764 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:20.471678 2563 scope.go:117] "RemoveContainer" containerID="d717b7df167903fe440573b6478eaadfc3ba6bb42f4a87d8339a6a616d3af4b1" Jul 17 19:20:20.471951 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:20.471938 2563 scope.go:117] "RemoveContainer" containerID="f9432ff197188d5aa2f26c3bf4220365e3581430ce71142f1f6887801665c337" Jul 17 19:20:20.472182 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:20:20.472160 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk_odh-ai-gateway-infra(2871553b-a031-4615-bca8-747cc11cd5c6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" Jul 17 19:20:21.499832 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:21.499797 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk"] Jul 17 19:20:21.643115 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:21.643091 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" Jul 17 19:20:21.765235 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:21.765144 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2871553b-a031-4615-bca8-747cc11cd5c6-maas-api-service-ca\") pod \"2871553b-a031-4615-bca8-747cc11cd5c6\" (UID: \"2871553b-a031-4615-bca8-747cc11cd5c6\") " Jul 17 19:20:21.765595 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:21.765566 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2871553b-a031-4615-bca8-747cc11cd5c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2871553b-a031-4615-bca8-747cc11cd5c6" (UID: "2871553b-a031-4615-bca8-747cc11cd5c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:20:21.865955 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:21.865921 2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2871553b-a031-4615-bca8-747cc11cd5c6-maas-api-service-ca\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:20:22.485603 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:22.485568 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" Jul 17 19:20:22.485781 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:22.485601 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk" event={"ID":"2871553b-a031-4615-bca8-747cc11cd5c6","Type":"ContainerDied","Data":"d2058fe1f474e7ce361674dbdb8cd54e3e867d66c2d113ca5fe3cac3870aa52e"} Jul 17 19:20:22.485781 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:22.485664 2563 scope.go:117] "RemoveContainer" containerID="f9432ff197188d5aa2f26c3bf4220365e3581430ce71142f1f6887801665c337" Jul 17 19:20:22.507417 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:22.507394 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk"] Jul 17 19:20:22.512959 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:22.512936 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b74adac0-zvtxk"] Jul 17 19:20:23.622724 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:20:23.622692 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" path="/var/lib/kubelet/pods/2871553b-a031-4615-bca8-747cc11cd5c6/volumes" Jul 17 19:22:50.230641 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.230608 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh"] Jul 17 19:22:50.231086 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231021 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" containerName="revoke-keys" Jul 17 19:22:50.231086 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231033 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" containerName="revoke-keys" Jul 17 19:22:50.231086 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231048 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" containerName="revoke-keys" Jul 17 19:22:50.231086 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231053 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" containerName="revoke-keys" Jul 17 19:22:50.231086 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231068 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" containerName="revoke-keys" Jul 17 19:22:50.231086 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231073 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" containerName="revoke-keys" Jul 17 19:22:50.231086 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231078 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" containerName="revoke-keys" Jul 17 19:22:50.231086 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231084 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" containerName="revoke-keys" Jul 17 19:22:50.231372 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231154 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" containerName="revoke-keys" Jul 17 19:22:50.231372 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231161 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" containerName="revoke-keys" Jul 17 19:22:50.231372 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231166 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" containerName="revoke-keys" Jul 17 19:22:50.231372 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231172 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" containerName="revoke-keys" Jul 17 19:22:50.231372 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231179 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" containerName="revoke-keys" Jul 17 19:22:50.231372 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.231185 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" containerName="revoke-keys" Jul 17 19:22:50.234327 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.234305 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.237756 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.237735 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-15f29d-openshift-default-dockercfg-t2ql7\"" Jul 17 19:22:50.246512 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.246481 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh"] Jul 17 19:22:50.322440 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.322398 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-envoy\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.322639 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.322524 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-token\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.322639 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.322601 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-certs\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.322639 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.322635 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-credential-socket\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.322821 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.322673 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ddm5\" (UniqueName: \"kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-kube-api-access-4ddm5\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.322821 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.322702 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-podinfo\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.322821 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.322750 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istiod-ca-cert\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.322954 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.322828 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-data\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.322954 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.322867 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-socket\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.423659 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.423624 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ddm5\" (UniqueName: \"kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-kube-api-access-4ddm5\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.423659 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.423660 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-podinfo\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.423915 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.423693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istiod-ca-cert\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.423915 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.423731 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-data\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.423915 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.423753 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-socket\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.423915 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.423792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-envoy\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.423915 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.423828 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-token\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.423915 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.423873 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-certs\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.423915 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.423893 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-credential-socket\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.424314 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.424239 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-socket\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.424314 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.424283 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-credential-socket\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.424429 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.424356 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-data\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.424429 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.424388 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-certs\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.424784 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.424759 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istiod-ca-cert\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.426500 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.426468 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-envoy\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.426826 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.426807 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-podinfo\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.431993 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.431969 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-token\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.432372 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.432352 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ddm5\" (UniqueName: \"kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-kube-api-access-4ddm5\") pod \"e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.547493 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.547382 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:50.697323 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.697296 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh"] Jul 17 19:22:50.699077 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:22:50.699042 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf93fe1_3fef_4b83_9806_dae8a979e47b.slice/crio-83a615e1b399cf9ff6c31635bed5c676feb858b8c6ddcfbda17a4518df3c4e1a WatchSource:0}: Error finding container 83a615e1b399cf9ff6c31635bed5c676feb858b8c6ddcfbda17a4518df3c4e1a: Status 404 returned error can't find the container with id 83a615e1b399cf9ff6c31635bed5c676feb858b8c6ddcfbda17a4518df3c4e1a Jul 17 19:22:50.701358 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.701318 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:22:50.701465 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.701406 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:22:50.701519 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:50.701473 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:22:51.288854 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:51.288813 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" event={"ID":"8bf93fe1-3fef-4b83-9806-dae8a979e47b","Type":"ContainerStarted","Data":"650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521"} Jul 17 19:22:51.289258 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:51.288901 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" event={"ID":"8bf93fe1-3fef-4b83-9806-dae8a979e47b","Type":"ContainerStarted","Data":"83a615e1b399cf9ff6c31635bed5c676feb858b8c6ddcfbda17a4518df3c4e1a"} Jul 17 19:22:51.311332 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:51.311276 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" podStartSLOduration=1.3112488629999999 podStartE2EDuration="1.311248863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:22:51.310087241 +0000 UTC m=+2988.213661127" watchObservedRunningTime="2026-07-17 19:22:51.311248863 +0000 UTC m=+2988.214822796" Jul 17 19:22:51.547890 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:51.547806 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:51.553080 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:51.553055 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:52.294767 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:52.294737 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:22:52.295729 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:22:52.295706 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:23:08.116286 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:08.115362 2563 patch_prober.go:28] interesting pod/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:40262->10.132.0.63:15021: read: connection reset by peer" start-of-body= Jul 17 19:23:08.116286 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:08.115439 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": read tcp 10.132.0.2:40262->10.132.0.63:15021: read: connection reset by peer" Jul 17 19:23:08.397414 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:08.397321 2563 generic.go:356] "Generic (PLEG): container finished" podID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerID="650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521" exitCode=0 Jul 17 19:23:08.397414 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:08.397395 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" event={"ID":"8bf93fe1-3fef-4b83-9806-dae8a979e47b","Type":"ContainerDied","Data":"650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521"} Jul 17 19:23:08.397841 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:08.397827 2563 scope.go:117] "RemoveContainer" containerID="650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521" Jul 17 19:23:08.398870 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:08.398843 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:23:08.398935 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:08.398910 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:23:08.398972 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:08.398942 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:23:09.404760 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:09.404725 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" event={"ID":"8bf93fe1-3fef-4b83-9806-dae8a979e47b","Type":"ContainerStarted","Data":"065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d"} Jul 17 19:23:09.548142 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:09.548103 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:23:09.548468 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:09.548419 2563 patch_prober.go:28] interesting pod/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 17 19:23:09.548602 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:09.548501 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 17 19:23:10.547811 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:10.547776 2563 patch_prober.go:28] interesting pod/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 17 19:23:10.548388 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:10.548354 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 17 19:23:11.547941 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:11.547898 2563 patch_prober.go:28] interesting pod/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 17 19:23:11.548432 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:11.547979 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 17 19:23:12.547942 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:12.547897 2563 patch_prober.go:28] interesting pod/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 17 19:23:12.548262 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:12.548228 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 17 19:23:13.552499 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:13.552470 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:23:13.552938 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:13.552921 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:23:13.553715 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:23:13.553694 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:25:04.350636 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:04.350602 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh"] Jul 17 19:25:04.351131 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:04.350870 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" containerID="cri-o://065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d" gracePeriod=30 Jul 17 19:25:09.617803 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.617740 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:25:09.708149 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708119 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-data\") pod \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " Jul 17 19:25:09.708338 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708183 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-envoy\") pod \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " Jul 17 19:25:09.708338 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708201 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-certs\") pod \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " Jul 17 19:25:09.708338 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708220 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-podinfo\") pod \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " Jul 17 19:25:09.708338 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708266 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-credential-socket\") pod \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " Jul 17 19:25:09.708602 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708376 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istiod-ca-cert\") pod \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " Jul 17 19:25:09.708602 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708422 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-token\") pod \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " Jul 17 19:25:09.708602 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708488 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-socket\") pod \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " Jul 17 19:25:09.708602 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708522 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ddm5\" (UniqueName: \"kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-kube-api-access-4ddm5\") pod \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\" (UID: \"8bf93fe1-3fef-4b83-9806-dae8a979e47b\") " Jul 17 19:25:09.708602 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708575 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8bf93fe1-3fef-4b83-9806-dae8a979e47b" (UID: "8bf93fe1-3fef-4b83-9806-dae8a979e47b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:25:09.708602 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708597 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8bf93fe1-3fef-4b83-9806-dae8a979e47b" (UID: "8bf93fe1-3fef-4b83-9806-dae8a979e47b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:25:09.708902 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708798 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8bf93fe1-3fef-4b83-9806-dae8a979e47b" (UID: "8bf93fe1-3fef-4b83-9806-dae8a979e47b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:25:09.708902 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708808 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8bf93fe1-3fef-4b83-9806-dae8a979e47b" (UID: "8bf93fe1-3fef-4b83-9806-dae8a979e47b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:25:09.709082 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708924 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istiod-ca-cert\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:25:09.709082 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708946 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-socket\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:25:09.709082 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708958 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-workload-certs\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:25:09.709082 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.708972 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-credential-socket\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:25:09.709082 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.709084 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-data" (OuterVolumeSpecName: "istio-data") pod "8bf93fe1-3fef-4b83-9806-dae8a979e47b" (UID: "8bf93fe1-3fef-4b83-9806-dae8a979e47b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:25:09.710661 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.710639 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8bf93fe1-3fef-4b83-9806-dae8a979e47b" (UID: "8bf93fe1-3fef-4b83-9806-dae8a979e47b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:25:09.710772 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.710674 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8bf93fe1-3fef-4b83-9806-dae8a979e47b" (UID: "8bf93fe1-3fef-4b83-9806-dae8a979e47b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 19:25:09.711272 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.711235 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-token" (OuterVolumeSpecName: "istio-token") pod "8bf93fe1-3fef-4b83-9806-dae8a979e47b" (UID: "8bf93fe1-3fef-4b83-9806-dae8a979e47b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:25:09.711272 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.711257 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-kube-api-access-4ddm5" (OuterVolumeSpecName: "kube-api-access-4ddm5") pod "8bf93fe1-3fef-4b83-9806-dae8a979e47b" (UID: "8bf93fe1-3fef-4b83-9806-dae8a979e47b"). InnerVolumeSpecName "kube-api-access-4ddm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:25:09.809955 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.809926 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-envoy\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:25:09.809955 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.809952 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-podinfo\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:25:09.809955 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.809961 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-token\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:25:09.810177 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.809971 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ddm5\" (UniqueName: \"kubernetes.io/projected/8bf93fe1-3fef-4b83-9806-dae8a979e47b-kube-api-access-4ddm5\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:25:09.810177 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:09.809979 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf93fe1-3fef-4b83-9806-dae8a979e47b-istio-data\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:25:10.067465 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.067402 2563 generic.go:356] "Generic (PLEG): container finished" podID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerID="065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d" exitCode=0 Jul 17 19:25:10.067652 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.067480 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" Jul 17 19:25:10.067652 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.067500 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" event={"ID":"8bf93fe1-3fef-4b83-9806-dae8a979e47b","Type":"ContainerDied","Data":"065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d"} Jul 17 19:25:10.067652 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.067576 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh" event={"ID":"8bf93fe1-3fef-4b83-9806-dae8a979e47b","Type":"ContainerDied","Data":"83a615e1b399cf9ff6c31635bed5c676feb858b8c6ddcfbda17a4518df3c4e1a"} Jul 17 19:25:10.067652 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.067613 2563 scope.go:117] "RemoveContainer" containerID="065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d" Jul 17 19:25:10.077591 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.077575 2563 scope.go:117] "RemoveContainer" containerID="650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521" Jul 17 19:25:10.085837 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.085812 2563 scope.go:117] "RemoveContainer" containerID="065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d" Jul 17 19:25:10.086080 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:25:10.086060 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d\": container with ID starting with 065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d not found: ID does not exist" containerID="065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d" Jul 17 19:25:10.086133 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.086090 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d"} err="failed to get container status \"065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d\": rpc error: code = NotFound desc = could not find container \"065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d\": container with ID starting with 065e8f834676fd24376ca6e3944e525ca08aa2fd5e7454952b5e010408e7173d not found: ID does not exist" Jul 17 19:25:10.086133 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.086107 2563 scope.go:117] "RemoveContainer" containerID="650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521" Jul 17 19:25:10.086332 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:25:10.086312 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521\": container with ID starting with 650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521 not found: ID does not exist" containerID="650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521" Jul 17 19:25:10.086397 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.086340 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521"} err="failed to get container status \"650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521\": rpc error: code = NotFound desc = could not find container \"650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521\": container with ID starting with 650a0353b07cd6075ef5e65522a14cec9c60378eab9ec9ddaf3272ab97bea521 not found: ID does not exist" Jul 17 19:25:10.089718 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.089695 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh"] Jul 17 19:25:10.094246 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:10.094228 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-15f29d-openshift-default-6d56597b6f-qr9fh"] Jul 17 19:25:11.623591 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:25:11.623560 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" path="/var/lib/kubelet/pods/8bf93fe1-3fef-4b83-9806-dae8a979e47b/volumes" Jul 17 19:26:21.088764 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.088731 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls"] Jul 17 19:26:21.089245 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.089177 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" containerName="revoke-keys" Jul 17 19:26:21.089245 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.089188 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd767773-8ee7-465d-b964-c7842db55e7f" containerName="revoke-keys" Jul 17 19:26:21.089245 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.089197 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" Jul 17 19:26:21.089245 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.089202 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" Jul 17 19:26:21.089245 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.089214 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" Jul 17 19:26:21.089245 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.089219 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" Jul 17 19:26:21.089245 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.089228 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" containerName="revoke-keys" Jul 17 19:26:21.089245 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.089234 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2871553b-a031-4615-bca8-747cc11cd5c6" containerName="revoke-keys" Jul 17 19:26:21.089520 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.089298 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" Jul 17 19:26:21.089520 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.089306 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf93fe1-3fef-4b83-9806-dae8a979e47b" containerName="istio-proxy" Jul 17 19:26:21.092564 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.092544 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.094830 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.094806 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-98e305-openshift-default-dockercfg-698fn\"" Jul 17 19:26:21.112882 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.112854 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls"] Jul 17 19:26:21.181381 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.181351 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-socket\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.181560 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.181396 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174c96a5-20bd-431a-afbe-11d9af43b135-istio-podinfo\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.181560 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.181507 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-envoy\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.181560 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.181541 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174c96a5-20bd-431a-afbe-11d9af43b135-istiod-ca-cert\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.181674 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.181570 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-credential-socket\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.181674 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.181605 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-certs\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.181674 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.181657 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-istio-token\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.181768 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.181699 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lztmc\" (UniqueName: \"kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-kube-api-access-lztmc\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.181768 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.181726 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-data\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283149 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283114 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-credential-socket\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283149 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283151 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-certs\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283351 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283187 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-istio-token\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283351 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283233 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lztmc\" (UniqueName: \"kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-kube-api-access-lztmc\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283351 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283263 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-data\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283351 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283321 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-socket\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283597 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283353 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174c96a5-20bd-431a-afbe-11d9af43b135-istio-podinfo\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283597 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283405 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-envoy\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283597 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174c96a5-20bd-431a-afbe-11d9af43b135-istiod-ca-cert\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283597 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283571 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-certs\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283785 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283748 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-socket\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283884 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283861 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-data\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.283884 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.283874 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-credential-socket\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.284139 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.284121 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174c96a5-20bd-431a-afbe-11d9af43b135-istiod-ca-cert\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.285878 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.285848 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-envoy\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.286198 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.286174 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174c96a5-20bd-431a-afbe-11d9af43b135-istio-podinfo\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.292502 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.292478 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lztmc\" (UniqueName: \"kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-kube-api-access-lztmc\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.292602 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.292440 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-istio-token\") pod \"e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.402509 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.402407 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:21.543512 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.543482 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls"] Jul 17 19:26:21.545303 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:26:21.545259 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174c96a5_20bd_431a_afbe_11d9af43b135.slice/crio-0f165545f9e89f4ae657c33d27a581b8359b90a62ad718bf4f72ea767a618d85 WatchSource:0}: Error finding container 0f165545f9e89f4ae657c33d27a581b8359b90a62ad718bf4f72ea767a618d85: Status 404 returned error can't find the container with id 0f165545f9e89f4ae657c33d27a581b8359b90a62ad718bf4f72ea767a618d85 Jul 17 19:26:21.547522 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.547490 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:26:21.547617 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.547553 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:26:21.547617 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:21.547583 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:26:22.450611 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:22.450574 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" event={"ID":"174c96a5-20bd-431a-afbe-11d9af43b135","Type":"ContainerStarted","Data":"7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035"} Jul 17 19:26:22.451093 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:22.450635 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" event={"ID":"174c96a5-20bd-431a-afbe-11d9af43b135","Type":"ContainerStarted","Data":"0f165545f9e89f4ae657c33d27a581b8359b90a62ad718bf4f72ea767a618d85"} Jul 17 19:26:22.471908 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:22.471860 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" podStartSLOduration=1.471844744 podStartE2EDuration="1.471844744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:26:22.46939184 +0000 UTC m=+3199.372965739" watchObservedRunningTime="2026-07-17 19:26:22.471844744 +0000 UTC m=+3199.375418695" Jul 17 19:26:23.403541 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:23.403507 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:23.408792 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:23.408768 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:23.456138 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:23.456108 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:23.456959 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:23.456940 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:37.534694 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:37.534661 2563 generic.go:356] "Generic (PLEG): container finished" podID="174c96a5-20bd-431a-afbe-11d9af43b135" containerID="7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035" exitCode=0 Jul 17 19:26:37.535118 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:37.534733 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" event={"ID":"174c96a5-20bd-431a-afbe-11d9af43b135","Type":"ContainerDied","Data":"7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035"} Jul 17 19:26:37.535163 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:37.535123 2563 scope.go:117] "RemoveContainer" containerID="7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035" Jul 17 19:26:37.536159 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:37.536132 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:26:37.536260 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:37.536207 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:26:37.536260 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:37.536239 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 17 19:26:38.456236 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:38.456185 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:38.543163 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:38.543128 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" event={"ID":"174c96a5-20bd-431a-afbe-11d9af43b135","Type":"ContainerStarted","Data":"4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329"} Jul 17 19:26:39.403189 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:39.403152 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:39.404226 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:39.404201 2563 patch_prober.go:28] interesting pod/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 17 19:26:39.404350 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:39.404252 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 17 19:26:40.403250 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:40.403213 2563 patch_prober.go:28] interesting pod/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 17 19:26:40.403668 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:40.403283 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 17 19:26:41.403057 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:41.403016 2563 patch_prober.go:28] interesting pod/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 17 19:26:41.403233 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:41.403087 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 17 19:26:42.407272 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:42.407241 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:42.407685 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:42.407607 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:26:42.408326 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:26:42.408308 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:29:50.302405 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.302367 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f"] Jul 17 19:29:50.306815 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.306791 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.309943 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.309918 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"default-dockercfg-5qjtz\"" Jul 17 19:29:50.309943 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.309934 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"sub-test-model-98e305-kserve-self-signed-certs\"" Jul 17 19:29:50.310151 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.309918 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"openshift-service-ca.crt\"" Jul 17 19:29:50.310151 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.309932 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-98e305\"/\"kube-root-ca.crt\"" Jul 17 19:29:50.313823 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.313799 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f"] Jul 17 19:29:50.412026 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.411989 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-home\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.412208 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.412043 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsmtt\" (UniqueName: \"kubernetes.io/projected/387f07bd-a7ad-42eb-90d2-6b3396464e11-kube-api-access-tsmtt\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.412208 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.412063 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-dshm\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.412208 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.412187 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387f07bd-a7ad-42eb-90d2-6b3396464e11-tls-certs\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.412317 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.412230 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-model-cache\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.412317 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.412301 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-tmp-dir\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.513053 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.513019 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387f07bd-a7ad-42eb-90d2-6b3396464e11-tls-certs\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.513281 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.513076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-model-cache\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.513281 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.513130 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-tmp-dir\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.513281 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.513173 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-home\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.513281 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.513222 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsmtt\" (UniqueName: \"kubernetes.io/projected/387f07bd-a7ad-42eb-90d2-6b3396464e11-kube-api-access-tsmtt\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.513281 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.513246 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-dshm\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.513705 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.513639 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-model-cache\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.513705 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.513679 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-home\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.513804 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.513741 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-tmp-dir\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.515637 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.515606 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-dshm\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.515889 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.515867 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387f07bd-a7ad-42eb-90d2-6b3396464e11-tls-certs\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.520718 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.520696 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsmtt\" (UniqueName: \"kubernetes.io/projected/387f07bd-a7ad-42eb-90d2-6b3396464e11-kube-api-access-tsmtt\") pod \"sub-test-model-98e305-kserve-85ff667cc9-d9h4f\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.617568 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.617476 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:29:50.766379 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:50.766328 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f"] Jul 17 19:29:50.769156 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:29:50.769123 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod387f07bd_a7ad_42eb_90d2_6b3396464e11.slice/crio-1ae4d1d32f53ad065a8bf30f8b1ba4fe56e61fa54437dbfd3cc304dc5ceeef81 WatchSource:0}: Error finding container 1ae4d1d32f53ad065a8bf30f8b1ba4fe56e61fa54437dbfd3cc304dc5ceeef81: Status 404 returned error can't find the container with id 1ae4d1d32f53ad065a8bf30f8b1ba4fe56e61fa54437dbfd3cc304dc5ceeef81 Jul 17 19:29:51.601663 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:51.601627 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" event={"ID":"387f07bd-a7ad-42eb-90d2-6b3396464e11","Type":"ContainerStarted","Data":"d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a"} Jul 17 19:29:51.602059 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:51.601677 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" event={"ID":"387f07bd-a7ad-42eb-90d2-6b3396464e11","Type":"ContainerStarted","Data":"1ae4d1d32f53ad065a8bf30f8b1ba4fe56e61fa54437dbfd3cc304dc5ceeef81"} Jul 17 19:29:51.631333 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:29:51.631281 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" podStartSLOduration=1.6312670150000002 podStartE2EDuration="1.631267015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 19:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:29:51.631199411 +0000 UTC m=+3408.534773288" watchObservedRunningTime="2026-07-17 19:29:51.631267015 +0000 UTC m=+3408.534840891" Jul 17 19:30:00.617891 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:30:00.617792 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:30:00.617891 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:30:00.617850 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:30:00.630943 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:30:00.630918 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:30:00.671390 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:30:00.671357 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:31:02.217204 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:02.217170 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f"] Jul 17 19:31:17.238003 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.237932 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" podUID="387f07bd-a7ad-42eb-90d2-6b3396464e11" containerName="main" containerID="cri-o://d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a" gracePeriod=45 Jul 17 19:31:17.509732 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.509709 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:31:17.580573 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.580535 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-model-cache\") pod \"387f07bd-a7ad-42eb-90d2-6b3396464e11\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " Jul 17 19:31:17.580776 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.580598 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsmtt\" (UniqueName: \"kubernetes.io/projected/387f07bd-a7ad-42eb-90d2-6b3396464e11-kube-api-access-tsmtt\") pod \"387f07bd-a7ad-42eb-90d2-6b3396464e11\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " Jul 17 19:31:17.580776 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.580668 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-home\") pod \"387f07bd-a7ad-42eb-90d2-6b3396464e11\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " Jul 17 19:31:17.580776 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.580696 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-tmp-dir\") pod \"387f07bd-a7ad-42eb-90d2-6b3396464e11\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " Jul 17 19:31:17.580776 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.580724 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387f07bd-a7ad-42eb-90d2-6b3396464e11-tls-certs\") pod \"387f07bd-a7ad-42eb-90d2-6b3396464e11\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " Jul 17 19:31:17.580776 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.580768 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-dshm\") pod \"387f07bd-a7ad-42eb-90d2-6b3396464e11\" (UID: \"387f07bd-a7ad-42eb-90d2-6b3396464e11\") " Jul 17 19:31:17.581015 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.580850 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-model-cache" (OuterVolumeSpecName: "model-cache") pod "387f07bd-a7ad-42eb-90d2-6b3396464e11" (UID: "387f07bd-a7ad-42eb-90d2-6b3396464e11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:31:17.581015 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.580943 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-home" (OuterVolumeSpecName: "home") pod "387f07bd-a7ad-42eb-90d2-6b3396464e11" (UID: "387f07bd-a7ad-42eb-90d2-6b3396464e11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:31:17.581015 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.580968 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "387f07bd-a7ad-42eb-90d2-6b3396464e11" (UID: "387f07bd-a7ad-42eb-90d2-6b3396464e11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:31:17.581224 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.581093 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-model-cache\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:31:17.581224 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.581105 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-home\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:31:17.581224 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.581121 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-tmp-dir\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:31:17.583085 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.583061 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387f07bd-a7ad-42eb-90d2-6b3396464e11-kube-api-access-tsmtt" (OuterVolumeSpecName: "kube-api-access-tsmtt") pod "387f07bd-a7ad-42eb-90d2-6b3396464e11" (UID: "387f07bd-a7ad-42eb-90d2-6b3396464e11"). InnerVolumeSpecName "kube-api-access-tsmtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:31:17.583182 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.583056 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-dshm" (OuterVolumeSpecName: "dshm") pod "387f07bd-a7ad-42eb-90d2-6b3396464e11" (UID: "387f07bd-a7ad-42eb-90d2-6b3396464e11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:31:17.583182 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.583138 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/387f07bd-a7ad-42eb-90d2-6b3396464e11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "387f07bd-a7ad-42eb-90d2-6b3396464e11" (UID: "387f07bd-a7ad-42eb-90d2-6b3396464e11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 19:31:17.682113 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.682073 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387f07bd-a7ad-42eb-90d2-6b3396464e11-tls-certs\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:31:17.682113 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.682103 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387f07bd-a7ad-42eb-90d2-6b3396464e11-dshm\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:31:17.682113 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:17.682112 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsmtt\" (UniqueName: \"kubernetes.io/projected/387f07bd-a7ad-42eb-90d2-6b3396464e11-kube-api-access-tsmtt\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:31:18.085991 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:18.085947 2563 generic.go:356] "Generic (PLEG): container finished" podID="387f07bd-a7ad-42eb-90d2-6b3396464e11" containerID="d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a" exitCode=0 Jul 17 19:31:18.086171 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:18.086015 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" Jul 17 19:31:18.086171 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:18.086014 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" event={"ID":"387f07bd-a7ad-42eb-90d2-6b3396464e11","Type":"ContainerDied","Data":"d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a"} Jul 17 19:31:18.086171 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:18.086121 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f" event={"ID":"387f07bd-a7ad-42eb-90d2-6b3396464e11","Type":"ContainerDied","Data":"1ae4d1d32f53ad065a8bf30f8b1ba4fe56e61fa54437dbfd3cc304dc5ceeef81"} Jul 17 19:31:18.086171 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:18.086141 2563 scope.go:117] "RemoveContainer" containerID="d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a" Jul 17 19:31:18.096007 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:18.095982 2563 scope.go:117] "RemoveContainer" containerID="d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a" Jul 17 19:31:18.096287 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:31:18.096265 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a\": container with ID starting with d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a not found: ID does not exist" containerID="d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a" Jul 17 19:31:18.096342 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:18.096300 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a"} err="failed to get container status \"d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a\": rpc error: code = NotFound desc = could not find container \"d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a\": container with ID starting with d78aab05ce3c0f446dc5a360ac3d7bee52bb6a26fa1520e3a95dfffb08c6609a not found: ID does not exist" Jul 17 19:31:18.102518 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:18.102494 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f"] Jul 17 19:31:18.108226 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:18.108201 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-98e305/sub-test-model-98e305-kserve-85ff667cc9-d9h4f"] Jul 17 19:31:19.623220 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:31:19.623184 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="387f07bd-a7ad-42eb-90d2-6b3396464e11" path="/var/lib/kubelet/pods/387f07bd-a7ad-42eb-90d2-6b3396464e11/volumes" Jul 17 19:32:29.871260 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:29.871215 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-139962106418608"] Jul 17 19:32:29.871755 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:29.871709 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="387f07bd-a7ad-42eb-90d2-6b3396464e11" containerName="main" Jul 17 19:32:29.871755 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:29.871724 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="387f07bd-a7ad-42eb-90d2-6b3396464e11" containerName="main" Jul 17 19:32:29.871831 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:29.871793 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="387f07bd-a7ad-42eb-90d2-6b3396464e11" containerName="main" Jul 17 19:32:29.875010 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:29.874992 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106418608" Jul 17 19:32:29.877369 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:29.877346 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v6jdh\"" Jul 17 19:32:29.883312 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:29.882866 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106418608"] Jul 17 19:32:29.980268 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:29.980222 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96z6\" (UniqueName: \"kubernetes.io/projected/0e5bb156-0469-419a-b5c1-dec75a862466-kube-api-access-f96z6\") pod \"test-curl-7735-139962106418608\" (UID: \"0e5bb156-0469-419a-b5c1-dec75a862466\") " pod="opendatahub/test-curl-7735-139962106418608" Jul 17 19:32:30.080824 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:30.080791 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96z6\" (UniqueName: \"kubernetes.io/projected/0e5bb156-0469-419a-b5c1-dec75a862466-kube-api-access-f96z6\") pod \"test-curl-7735-139962106418608\" (UID: \"0e5bb156-0469-419a-b5c1-dec75a862466\") " pod="opendatahub/test-curl-7735-139962106418608" Jul 17 19:32:30.089041 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:30.089011 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96z6\" (UniqueName: \"kubernetes.io/projected/0e5bb156-0469-419a-b5c1-dec75a862466-kube-api-access-f96z6\") pod \"test-curl-7735-139962106418608\" (UID: \"0e5bb156-0469-419a-b5c1-dec75a862466\") " pod="opendatahub/test-curl-7735-139962106418608" Jul 17 19:32:30.193721 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:30.193694 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106418608" Jul 17 19:32:30.328136 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:30.328103 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106418608"] Jul 17 19:32:30.329707 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:32:30.329666 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e5bb156_0469_419a_b5c1_dec75a862466.slice/crio-3dcea9765667d001d695279b0c55a68010df109490874dcb2f0955d3930ea126 WatchSource:0}: Error finding container 3dcea9765667d001d695279b0c55a68010df109490874dcb2f0955d3930ea126: Status 404 returned error can't find the container with id 3dcea9765667d001d695279b0c55a68010df109490874dcb2f0955d3930ea126 Jul 17 19:32:30.331195 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:30.331174 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 19:32:30.486354 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:30.486268 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106418608" event={"ID":"0e5bb156-0469-419a-b5c1-dec75a862466","Type":"ContainerStarted","Data":"3dcea9765667d001d695279b0c55a68010df109490874dcb2f0955d3930ea126"} Jul 17 19:32:31.495051 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:31.495017 2563 generic.go:356] "Generic (PLEG): container finished" podID="0e5bb156-0469-419a-b5c1-dec75a862466" containerID="f1559aa002513c90c0ffa6fc1789e4c0f541c11f7e45af1489bc636c63bfbbdf" exitCode=0 Jul 17 19:32:31.495421 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:31.495098 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106418608" event={"ID":"0e5bb156-0469-419a-b5c1-dec75a862466","Type":"ContainerDied","Data":"f1559aa002513c90c0ffa6fc1789e4c0f541c11f7e45af1489bc636c63bfbbdf"} Jul 17 19:32:32.641011 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:32.640988 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106418608" Jul 17 19:32:32.681241 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:32.681209 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7735-139962106418608_0e5bb156-0469-419a-b5c1-dec75a862466/test-curl-7735-139962106418608/0.log" Jul 17 19:32:32.707134 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:32.707106 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-139962106418608"] Jul 17 19:32:32.712461 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:32.712423 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-139962106418608"] Jul 17 19:32:32.807788 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:32.807709 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f96z6\" (UniqueName: \"kubernetes.io/projected/0e5bb156-0469-419a-b5c1-dec75a862466-kube-api-access-f96z6\") pod \"0e5bb156-0469-419a-b5c1-dec75a862466\" (UID: \"0e5bb156-0469-419a-b5c1-dec75a862466\") " Jul 17 19:32:32.809759 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:32.809727 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5bb156-0469-419a-b5c1-dec75a862466-kube-api-access-f96z6" (OuterVolumeSpecName: "kube-api-access-f96z6") pod "0e5bb156-0469-419a-b5c1-dec75a862466" (UID: "0e5bb156-0469-419a-b5c1-dec75a862466"). InnerVolumeSpecName "kube-api-access-f96z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:32:32.908423 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:32.908387 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f96z6\" (UniqueName: \"kubernetes.io/projected/0e5bb156-0469-419a-b5c1-dec75a862466-kube-api-access-f96z6\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:32:33.507769 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:33.507739 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106418608" Jul 17 19:32:33.507945 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:33.507784 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dcea9765667d001d695279b0c55a68010df109490874dcb2f0955d3930ea126" Jul 17 19:32:33.516996 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:32:33.516969 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-139962106418608\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="0e5bb156-0469-419a-b5c1-dec75a862466" pod="opendatahub/test-curl-7735-139962106418608" Jul 17 19:32:33.621667 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:32:33.621630 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-139962106418608\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="0e5bb156-0469-419a-b5c1-dec75a862466" pod="opendatahub/test-curl-7735-139962106418608" Jul 17 19:32:33.623098 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:33.623067 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e5bb156-0469-419a-b5c1-dec75a862466" path="/var/lib/kubelet/pods/0e5bb156-0469-419a-b5c1-dec75a862466/volumes" Jul 17 19:32:35.605016 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.604981 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-139962106361152"] Jul 17 19:32:35.605498 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.605484 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e5bb156-0469-419a-b5c1-dec75a862466" containerName="test-curl-7735-139962106418608" Jul 17 19:32:35.605549 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.605500 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e5bb156-0469-419a-b5c1-dec75a862466" containerName="test-curl-7735-139962106418608" Jul 17 19:32:35.605584 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.605575 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e5bb156-0469-419a-b5c1-dec75a862466" containerName="test-curl-7735-139962106418608" Jul 17 19:32:35.608722 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.608708 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106361152" Jul 17 19:32:35.611025 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.610992 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v6jdh\"" Jul 17 19:32:35.613371 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.613351 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106361152"] Jul 17 19:32:35.628302 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.628273 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvpnk\" (UniqueName: \"kubernetes.io/projected/00f36de0-8aba-4ef9-864c-b97f9791042c-kube-api-access-vvpnk\") pod \"test-curl-7735-139962106361152\" (UID: \"00f36de0-8aba-4ef9-864c-b97f9791042c\") " pod="opendatahub/test-curl-7735-139962106361152" Jul 17 19:32:35.729157 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.729118 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvpnk\" (UniqueName: \"kubernetes.io/projected/00f36de0-8aba-4ef9-864c-b97f9791042c-kube-api-access-vvpnk\") pod \"test-curl-7735-139962106361152\" (UID: \"00f36de0-8aba-4ef9-864c-b97f9791042c\") " pod="opendatahub/test-curl-7735-139962106361152" Jul 17 19:32:35.737355 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.737328 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvpnk\" (UniqueName: \"kubernetes.io/projected/00f36de0-8aba-4ef9-864c-b97f9791042c-kube-api-access-vvpnk\") pod \"test-curl-7735-139962106361152\" (UID: \"00f36de0-8aba-4ef9-864c-b97f9791042c\") " pod="opendatahub/test-curl-7735-139962106361152" Jul 17 19:32:35.919521 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:35.919489 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106361152" Jul 17 19:32:36.054949 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:36.054903 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106361152"] Jul 17 19:32:36.057484 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:32:36.057428 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00f36de0_8aba_4ef9_864c_b97f9791042c.slice/crio-df9b048656042d049fc610bf4669e519bfea6469a76feb637714d15fc977094f WatchSource:0}: Error finding container df9b048656042d049fc610bf4669e519bfea6469a76feb637714d15fc977094f: Status 404 returned error can't find the container with id df9b048656042d049fc610bf4669e519bfea6469a76feb637714d15fc977094f Jul 17 19:32:36.525067 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:36.525033 2563 generic.go:356] "Generic (PLEG): container finished" podID="00f36de0-8aba-4ef9-864c-b97f9791042c" containerID="a1428cbc73cbf5ae13940f2406d7131fb2c05ad7b8511148111b1a08cae05f53" exitCode=0 Jul 17 19:32:36.525217 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:36.525076 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106361152" event={"ID":"00f36de0-8aba-4ef9-864c-b97f9791042c","Type":"ContainerDied","Data":"a1428cbc73cbf5ae13940f2406d7131fb2c05ad7b8511148111b1a08cae05f53"} Jul 17 19:32:36.525217 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:36.525117 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106361152" event={"ID":"00f36de0-8aba-4ef9-864c-b97f9791042c","Type":"ContainerStarted","Data":"df9b048656042d049fc610bf4669e519bfea6469a76feb637714d15fc977094f"} Jul 17 19:32:37.671193 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:37.671169 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106361152" Jul 17 19:32:37.691057 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:37.691035 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7735-139962106361152_00f36de0-8aba-4ef9-864c-b97f9791042c/test-curl-7735-139962106361152/0.log" Jul 17 19:32:37.714955 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:37.714924 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-139962106361152"] Jul 17 19:32:37.718771 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:37.718744 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-139962106361152"] Jul 17 19:32:37.745913 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:37.745887 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvpnk\" (UniqueName: \"kubernetes.io/projected/00f36de0-8aba-4ef9-864c-b97f9791042c-kube-api-access-vvpnk\") pod \"00f36de0-8aba-4ef9-864c-b97f9791042c\" (UID: \"00f36de0-8aba-4ef9-864c-b97f9791042c\") " Jul 17 19:32:37.747997 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:37.747975 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f36de0-8aba-4ef9-864c-b97f9791042c-kube-api-access-vvpnk" (OuterVolumeSpecName: "kube-api-access-vvpnk") pod "00f36de0-8aba-4ef9-864c-b97f9791042c" (UID: "00f36de0-8aba-4ef9-864c-b97f9791042c"). InnerVolumeSpecName "kube-api-access-vvpnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:32:37.847010 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:37.846928 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvpnk\" (UniqueName: \"kubernetes.io/projected/00f36de0-8aba-4ef9-864c-b97f9791042c-kube-api-access-vvpnk\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:32:38.537554 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:38.537524 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df9b048656042d049fc610bf4669e519bfea6469a76feb637714d15fc977094f" Jul 17 19:32:38.537554 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:38.537525 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106361152" Jul 17 19:32:38.547576 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:32:38.547546 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-139962106361152\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="00f36de0-8aba-4ef9-864c-b97f9791042c" pod="opendatahub/test-curl-7735-139962106361152" Jul 17 19:32:39.623416 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:39.623384 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f36de0-8aba-4ef9-864c-b97f9791042c" path="/var/lib/kubelet/pods/00f36de0-8aba-4ef9-864c-b97f9791042c/volumes" Jul 17 19:32:40.609100 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.609056 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7735-139962106362160"] Jul 17 19:32:40.609615 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.609595 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f36de0-8aba-4ef9-864c-b97f9791042c" containerName="test-curl-7735-139962106361152" Jul 17 19:32:40.609615 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.609617 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f36de0-8aba-4ef9-864c-b97f9791042c" containerName="test-curl-7735-139962106361152" Jul 17 19:32:40.609794 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.609771 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f36de0-8aba-4ef9-864c-b97f9791042c" containerName="test-curl-7735-139962106361152" Jul 17 19:32:40.613047 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.613025 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106362160" Jul 17 19:32:40.615200 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.615183 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v6jdh\"" Jul 17 19:32:40.619738 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.619700 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106362160"] Jul 17 19:32:40.669423 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.669395 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4kpc\" (UniqueName: \"kubernetes.io/projected/48a33718-179d-409e-8df7-07740896cdef-kube-api-access-f4kpc\") pod \"test-curl-7735-139962106362160\" (UID: \"48a33718-179d-409e-8df7-07740896cdef\") " pod="opendatahub/test-curl-7735-139962106362160" Jul 17 19:32:40.770232 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.770201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4kpc\" (UniqueName: \"kubernetes.io/projected/48a33718-179d-409e-8df7-07740896cdef-kube-api-access-f4kpc\") pod \"test-curl-7735-139962106362160\" (UID: \"48a33718-179d-409e-8df7-07740896cdef\") " pod="opendatahub/test-curl-7735-139962106362160" Jul 17 19:32:40.778126 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.778099 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4kpc\" (UniqueName: \"kubernetes.io/projected/48a33718-179d-409e-8df7-07740896cdef-kube-api-access-f4kpc\") pod \"test-curl-7735-139962106362160\" (UID: \"48a33718-179d-409e-8df7-07740896cdef\") " pod="opendatahub/test-curl-7735-139962106362160" Jul 17 19:32:40.924400 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:40.924369 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106362160" Jul 17 19:32:41.057886 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:41.057858 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7735-139962106362160"] Jul 17 19:32:41.059520 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:32:41.059495 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48a33718_179d_409e_8df7_07740896cdef.slice/crio-d134533a4876cb686d515a7b336b468c151a9207215a8a423a0b3661f1195d66 WatchSource:0}: Error finding container d134533a4876cb686d515a7b336b468c151a9207215a8a423a0b3661f1195d66: Status 404 returned error can't find the container with id d134533a4876cb686d515a7b336b468c151a9207215a8a423a0b3661f1195d66 Jul 17 19:32:41.557110 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:41.557066 2563 generic.go:356] "Generic (PLEG): container finished" podID="48a33718-179d-409e-8df7-07740896cdef" containerID="3fa8e8fce715fde64f526acd507e8664d79dac55b5772faa5f1dfa3d4d7f722a" exitCode=0 Jul 17 19:32:41.557110 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:41.557107 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106362160" event={"ID":"48a33718-179d-409e-8df7-07740896cdef","Type":"ContainerDied","Data":"3fa8e8fce715fde64f526acd507e8664d79dac55b5772faa5f1dfa3d4d7f722a"} Jul 17 19:32:41.557299 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:41.557143 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7735-139962106362160" event={"ID":"48a33718-179d-409e-8df7-07740896cdef","Type":"ContainerStarted","Data":"d134533a4876cb686d515a7b336b468c151a9207215a8a423a0b3661f1195d66"} Jul 17 19:32:42.705848 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:42.705826 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106362160" Jul 17 19:32:42.724212 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:42.724161 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7735-139962106362160_48a33718-179d-409e-8df7-07740896cdef/test-curl-7735-139962106362160/0.log" Jul 17 19:32:42.747737 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:42.747701 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7735-139962106362160"] Jul 17 19:32:42.754352 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:42.754329 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7735-139962106362160"] Jul 17 19:32:42.788316 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:42.788293 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4kpc\" (UniqueName: \"kubernetes.io/projected/48a33718-179d-409e-8df7-07740896cdef-kube-api-access-f4kpc\") pod \"48a33718-179d-409e-8df7-07740896cdef\" (UID: \"48a33718-179d-409e-8df7-07740896cdef\") " Jul 17 19:32:42.790494 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:42.790470 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a33718-179d-409e-8df7-07740896cdef-kube-api-access-f4kpc" (OuterVolumeSpecName: "kube-api-access-f4kpc") pod "48a33718-179d-409e-8df7-07740896cdef" (UID: "48a33718-179d-409e-8df7-07740896cdef"). InnerVolumeSpecName "kube-api-access-f4kpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:32:42.889854 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:42.889781 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4kpc\" (UniqueName: \"kubernetes.io/projected/48a33718-179d-409e-8df7-07740896cdef-kube-api-access-f4kpc\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:32:43.569636 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:43.569606 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d134533a4876cb686d515a7b336b468c151a9207215a8a423a0b3661f1195d66" Jul 17 19:32:43.569636 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:43.569625 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7735-139962106362160" Jul 17 19:32:43.578807 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:32:43.578780 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-139962106362160\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="48a33718-179d-409e-8df7-07740896cdef" pod="opendatahub/test-curl-7735-139962106362160" Jul 17 19:32:43.620853 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:32:43.620823 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7735-139962106362160\" is forbidden: User \"system:node:ip-10-0-132-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-220.ec2.internal' and this object" podUID="48a33718-179d-409e-8df7-07740896cdef" pod="opendatahub/test-curl-7735-139962106362160" Jul 17 19:32:43.623196 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:32:43.623176 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48a33718-179d-409e-8df7-07740896cdef" path="/var/lib/kubelet/pods/48a33718-179d-409e-8df7-07740896cdef/volumes" Jul 17 19:33:11.451361 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:11.451263 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls"] Jul 17 19:33:11.451851 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:11.451590 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" containerID="cri-o://4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329" gracePeriod=30 Jul 17 19:33:12.409765 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:12.409724 2563 patch_prober.go:28] interesting pod/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 19:33:12.409933 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:12.409790 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 19:33:16.721836 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.721776 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:33:16.724938 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.724917 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-istio-token\") pod \"174c96a5-20bd-431a-afbe-11d9af43b135\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " Jul 17 19:33:16.725017 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.724991 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lztmc\" (UniqueName: \"kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-kube-api-access-lztmc\") pod \"174c96a5-20bd-431a-afbe-11d9af43b135\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " Jul 17 19:33:16.725075 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725025 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-socket\") pod \"174c96a5-20bd-431a-afbe-11d9af43b135\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " Jul 17 19:33:16.725075 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725050 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-credential-socket\") pod \"174c96a5-20bd-431a-afbe-11d9af43b135\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " Jul 17 19:33:16.725178 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725086 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-data\") pod \"174c96a5-20bd-431a-afbe-11d9af43b135\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " Jul 17 19:33:16.725178 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725117 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174c96a5-20bd-431a-afbe-11d9af43b135-istiod-ca-cert\") pod \"174c96a5-20bd-431a-afbe-11d9af43b135\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " Jul 17 19:33:16.725178 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725149 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174c96a5-20bd-431a-afbe-11d9af43b135-istio-podinfo\") pod \"174c96a5-20bd-431a-afbe-11d9af43b135\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " Jul 17 19:33:16.725178 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725170 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-envoy\") pod \"174c96a5-20bd-431a-afbe-11d9af43b135\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " Jul 17 19:33:16.725370 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725205 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-certs\") pod \"174c96a5-20bd-431a-afbe-11d9af43b135\" (UID: \"174c96a5-20bd-431a-afbe-11d9af43b135\") " Jul 17 19:33:16.725370 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725322 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "174c96a5-20bd-431a-afbe-11d9af43b135" (UID: "174c96a5-20bd-431a-afbe-11d9af43b135"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:33:16.725498 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725408 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "174c96a5-20bd-431a-afbe-11d9af43b135" (UID: "174c96a5-20bd-431a-afbe-11d9af43b135"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:33:16.725569 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725530 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174c96a5-20bd-431a-afbe-11d9af43b135-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "174c96a5-20bd-431a-afbe-11d9af43b135" (UID: "174c96a5-20bd-431a-afbe-11d9af43b135"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 19:33:16.725624 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725574 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-socket\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:33:16.725678 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725621 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-credential-socket\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:33:16.725678 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.725662 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "174c96a5-20bd-431a-afbe-11d9af43b135" (UID: "174c96a5-20bd-431a-afbe-11d9af43b135"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:33:16.726356 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.726325 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-data" (OuterVolumeSpecName: "istio-data") pod "174c96a5-20bd-431a-afbe-11d9af43b135" (UID: "174c96a5-20bd-431a-afbe-11d9af43b135"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:33:16.727303 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.727273 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-kube-api-access-lztmc" (OuterVolumeSpecName: "kube-api-access-lztmc") pod "174c96a5-20bd-431a-afbe-11d9af43b135" (UID: "174c96a5-20bd-431a-afbe-11d9af43b135"). InnerVolumeSpecName "kube-api-access-lztmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:33:16.727411 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.727303 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-istio-token" (OuterVolumeSpecName: "istio-token") pod "174c96a5-20bd-431a-afbe-11d9af43b135" (UID: "174c96a5-20bd-431a-afbe-11d9af43b135"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 19:33:16.727924 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.727907 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "174c96a5-20bd-431a-afbe-11d9af43b135" (UID: "174c96a5-20bd-431a-afbe-11d9af43b135"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 19:33:16.728105 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.728088 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/174c96a5-20bd-431a-afbe-11d9af43b135-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "174c96a5-20bd-431a-afbe-11d9af43b135" (UID: "174c96a5-20bd-431a-afbe-11d9af43b135"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 17 19:33:16.753123 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.753091 2563 generic.go:356] "Generic (PLEG): container finished" podID="174c96a5-20bd-431a-afbe-11d9af43b135" containerID="4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329" exitCode=0 Jul 17 19:33:16.753293 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.753142 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" Jul 17 19:33:16.753293 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.753175 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" event={"ID":"174c96a5-20bd-431a-afbe-11d9af43b135","Type":"ContainerDied","Data":"4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329"} Jul 17 19:33:16.753293 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.753224 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls" event={"ID":"174c96a5-20bd-431a-afbe-11d9af43b135","Type":"ContainerDied","Data":"0f165545f9e89f4ae657c33d27a581b8359b90a62ad718bf4f72ea767a618d85"} Jul 17 19:33:16.753293 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.753245 2563 scope.go:117] "RemoveContainer" containerID="4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329" Jul 17 19:33:16.767865 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.767843 2563 scope.go:117] "RemoveContainer" containerID="7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035" Jul 17 19:33:16.777474 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.777406 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls"] Jul 17 19:33:16.777671 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.777505 2563 scope.go:117] "RemoveContainer" containerID="4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329" Jul 17 19:33:16.777827 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:33:16.777807 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329\": container with ID starting with 4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329 not found: ID does not exist" containerID="4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329" Jul 17 19:33:16.777867 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.777837 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329"} err="failed to get container status \"4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329\": rpc error: code = NotFound desc = could not find container \"4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329\": container with ID starting with 4db145fc452b8697f417f918a50deb6c780c670bb9763ab9f7ff26a0917f7329 not found: ID does not exist" Jul 17 19:33:16.777867 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.777855 2563 scope.go:117] "RemoveContainer" containerID="7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035" Jul 17 19:33:16.778119 ip-10-0-132-220 kubenswrapper[2563]: E0717 19:33:16.778100 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035\": container with ID starting with 7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035 not found: ID does not exist" containerID="7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035" Jul 17 19:33:16.778158 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.778127 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035"} err="failed to get container status \"7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035\": rpc error: code = NotFound desc = could not find container \"7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035\": container with ID starting with 7da943ebf986934e7328112ea0da251eb17423778d25fbddb4281196d366f035 not found: ID does not exist" Jul 17 19:33:16.781606 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.781586 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-98e305-openshift-default-85587fc8c9-v4rls"] Jul 17 19:33:16.826797 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.826770 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-data\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:33:16.826797 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.826791 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/174c96a5-20bd-431a-afbe-11d9af43b135-istiod-ca-cert\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:33:16.826797 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.826799 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/174c96a5-20bd-431a-afbe-11d9af43b135-istio-podinfo\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:33:16.826996 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.826807 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-istio-envoy\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:33:16.826996 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.826816 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/174c96a5-20bd-431a-afbe-11d9af43b135-workload-certs\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:33:16.826996 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.826823 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-istio-token\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:33:16.826996 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:16.826831 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lztmc\" (UniqueName: \"kubernetes.io/projected/174c96a5-20bd-431a-afbe-11d9af43b135-kube-api-access-lztmc\") on node \"ip-10-0-132-220.ec2.internal\" DevicePath \"\"" Jul 17 19:33:17.624505 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:17.624470 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" path="/var/lib/kubelet/pods/174c96a5-20bd-431a-afbe-11d9af43b135/volumes" Jul 17 19:33:38.258527 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:38.258489 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f88c8798d-7vq99_b6f21b3f-e568-40e0-a58e-cef553291f80/authorino/0.log" Jul 17 19:33:43.521019 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:43.520974 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5c8qs_0db86b23-a5c6-4919-b36b-d05d33d44e6c/manager/0.log" Jul 17 19:33:43.645394 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:43.645364 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7bqrm_7a768798-edd2-49f7-81e6-8eb5d880c6f2/manager/0.log" Jul 17 19:33:43.887752 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:43.887668 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zqxbn_1d960e12-58ba-4994-b607-2e80aff93084/server/0.log" Jul 17 19:33:44.012748 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:44.012709 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6prfl_1aa7760b-8a78-4310-b9ea-3482fcc94ba8/manager/0.log" Jul 17 19:33:44.247632 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:44.247602 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-qdzll_e4467a8e-8f06-4c08-890d-514366edb029/copy-manifests/0.log" Jul 17 19:33:44.256288 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:44.256264 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-qdzll_e4467a8e-8f06-4c08-890d-514366edb029/manager/0.log" Jul 17 19:33:47.603789 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:47.603758 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f88c8798d-7vq99_b6f21b3f-e568-40e0-a58e-cef553291f80/authorino/0.log" Jul 17 19:33:47.716271 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:47.716240 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7hd27_4ed78ed4-deb8-4da4-b63e-d520146d5d87/manager/0.log" Jul 17 19:33:47.820240 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:47.820209 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-hlb98_821d5ca9-2299-4fdc-92d4-635f95617da3/manager/0.log" Jul 17 19:33:48.050321 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:48.050288 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hzcmh_f0b5eef6-8b4f-40df-9188-2bd00ca24189/registry-server/0.log" Jul 17 19:33:48.170202 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:48.170172 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-fl9s6_37c99017-dcf8-4054-bc3e-6de71d9d5141/manager/0.log" Jul 17 19:33:48.406814 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:48.406777 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ljdjz_0f0f928b-1cc2-416d-934c-02fc6901e9f8/manager/0.log" Jul 17 19:33:48.759735 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:48.759709 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf484jg9_5b252de0-39c6-4fd8-9bf6-a9cee59a6c33/istio-proxy/0.log" Jul 17 19:33:48.977812 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:48.977787 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864488b8fd-6g5lc_05328159-9607-4cc5-898b-25388d78633b/kube-auth-proxy/0.log" Jul 17 19:33:49.211713 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:49.211680 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rvs2d_27310e84-6e5b-42d5-ba9a-ea4920dc6071/istio-proxy/1.log" Jul 17 19:33:49.328265 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:49.328233 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-srdlz_962e09eb-d2ca-41e4-ae57-16cfb445573c/payload-pre-processing/0.log" Jul 17 19:33:49.448336 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:49.448306 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-dc72j_8e68af24-81ea-4746-9d79-3ad158296036/payload-processing/0.log" Jul 17 19:33:49.565632 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:49.565548 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57c488db8b-cxf75_1c73c210-fc62-4860-93e4-c352a8021182/router/0.log" Jul 17 19:33:50.007299 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:50.007264 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jpknd_1153df46-a624-48d4-bbd1-a3d2154479d3/main/0.log" Jul 17 19:33:50.118280 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:50.118247 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-vmphm_be1afb96-0ad6-4ef5-abe2-3fc54bd54674/main/0.log" Jul 17 19:33:50.446768 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:50.446736 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-dnbnj_7e0b7e4b-1dcd-494d-84e8-84e5203783b0/main/0.log" Jul 17 19:33:58.828841 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:58.828809 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kccpg_0f74a126-e2b2-4930-91f5-d8fab73ba4af/global-pull-secret-syncer/0.log" Jul 17 19:33:58.969755 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:58.969716 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rkn6w_69e728c7-854c-4ff2-a3e6-648a15b9c47f/konnectivity-agent/0.log" Jul 17 19:33:59.033224 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:33:59.033173 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-220.ec2.internal_6ece1844396d1b4d185e7c0e925cd74d/haproxy/0.log" Jul 17 19:34:04.193287 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:04.193250 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f88c8798d-7vq99_b6f21b3f-e568-40e0-a58e-cef553291f80/authorino/0.log" Jul 17 19:34:04.239884 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:04.239850 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7hd27_4ed78ed4-deb8-4da4-b63e-d520146d5d87/manager/0.log" Jul 17 19:34:04.266986 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:04.266948 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-hlb98_821d5ca9-2299-4fdc-92d4-635f95617da3/manager/0.log" Jul 17 19:34:04.355154 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:04.355079 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-hzcmh_f0b5eef6-8b4f-40df-9188-2bd00ca24189/registry-server/0.log" Jul 17 19:34:04.558819 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:04.558786 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-fl9s6_37c99017-dcf8-4054-bc3e-6de71d9d5141/manager/0.log" Jul 17 19:34:04.679756 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:04.679717 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-ljdjz_0f0f928b-1cc2-416d-934c-02fc6901e9f8/manager/0.log" Jul 17 19:34:06.707123 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:06.707084 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-75c8cfd894-7jmvl_02db1c12-fbff-4d59-a0ae-c0935d4694a0/metrics-server/0.log" Jul 17 19:34:06.741698 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:06.741673 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-f58fc_c86ee112-e81f-4e44-bb11-a04adea1c35b/monitoring-plugin/0.log" Jul 17 19:34:06.922739 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:06.922703 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvrnl_6860e99a-1e8f-4a35-8ada-18e31b298764/node-exporter/0.log" Jul 17 19:34:06.945352 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:06.945328 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvrnl_6860e99a-1e8f-4a35-8ada-18e31b298764/kube-rbac-proxy/0.log" Jul 17 19:34:06.965589 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:06.965521 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvrnl_6860e99a-1e8f-4a35-8ada-18e31b298764/init-textfile/0.log" Jul 17 19:34:06.991933 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:06.991907 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-kjqtg_0b075b8a-e5b0-4008-ae0e-c2a383482e5d/kube-rbac-proxy-main/0.log" Jul 17 19:34:07.012644 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.012616 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-kjqtg_0b075b8a-e5b0-4008-ae0e-c2a383482e5d/kube-rbac-proxy-self/0.log" Jul 17 19:34:07.035923 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.035887 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-kjqtg_0b075b8a-e5b0-4008-ae0e-c2a383482e5d/openshift-state-metrics/0.log" Jul 17 19:34:07.232575 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.232487 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-cx76z_d2dfa1cb-2b67-4ba2-b271-30f06b605820/prometheus-operator/0.log" Jul 17 19:34:07.247859 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.247831 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-cx76z_d2dfa1cb-2b67-4ba2-b271-30f06b605820/kube-rbac-proxy/0.log" Jul 17 19:34:07.274438 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.274414 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-qf47l_e606a35c-d2e7-41c3-85eb-b24a1aa29e97/prometheus-operator-admission-webhook/0.log" Jul 17 19:34:07.390688 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.390657 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7475d87648-pf47s_3d5c6871-9db6-451d-bcb3-ddc377c9d37b/thanos-query/0.log" Jul 17 19:34:07.411228 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.411203 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7475d87648-pf47s_3d5c6871-9db6-451d-bcb3-ddc377c9d37b/kube-rbac-proxy-web/0.log" Jul 17 19:34:07.431729 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.431697 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7475d87648-pf47s_3d5c6871-9db6-451d-bcb3-ddc377c9d37b/kube-rbac-proxy/0.log" Jul 17 19:34:07.453567 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.453544 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7475d87648-pf47s_3d5c6871-9db6-451d-bcb3-ddc377c9d37b/prom-label-proxy/0.log" Jul 17 19:34:07.486698 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.486590 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7475d87648-pf47s_3d5c6871-9db6-451d-bcb3-ddc377c9d37b/kube-rbac-proxy-rules/0.log" Jul 17 19:34:07.511385 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:07.511355 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7475d87648-pf47s_3d5c6871-9db6-451d-bcb3-ddc377c9d37b/kube-rbac-proxy-metrics/0.log" Jul 17 19:34:08.590532 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.590500 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd"] Jul 17 19:34:08.590924 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.590909 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a33718-179d-409e-8df7-07740896cdef" containerName="test-curl-7735-139962106362160" Jul 17 19:34:08.590969 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.590926 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a33718-179d-409e-8df7-07740896cdef" containerName="test-curl-7735-139962106362160" Jul 17 19:34:08.590969 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.590949 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" Jul 17 19:34:08.590969 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.590955 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" Jul 17 19:34:08.590969 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.590962 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" Jul 17 19:34:08.590969 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.590967 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" Jul 17 19:34:08.591128 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.591030 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" Jul 17 19:34:08.591128 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.591038 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="174c96a5-20bd-431a-afbe-11d9af43b135" containerName="istio-proxy" Jul 17 19:34:08.591128 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.591045 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a33718-179d-409e-8df7-07740896cdef" containerName="test-curl-7735-139962106362160" Jul 17 19:34:08.595301 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.595286 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.597316 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.597296 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtqk6\"/\"openshift-service-ca.crt\"" Jul 17 19:34:08.597425 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.597348 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dtqk6\"/\"default-dockercfg-n4gpm\"" Jul 17 19:34:08.598085 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.598070 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dtqk6\"/\"kube-root-ca.crt\"" Jul 17 19:34:08.602077 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.602049 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd"] Jul 17 19:34:08.649462 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.649414 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-ghl9n_d1af3565-93a3-4ce2-933d-c527fc2102ca/networking-console-plugin/0.log" Jul 17 19:34:08.709639 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.709605 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-lib-modules\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.709782 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.709653 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44qc7\" (UniqueName: \"kubernetes.io/projected/36398fbd-60d9-4867-b360-5a17fc3aba2e-kube-api-access-44qc7\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.709782 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.709746 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-podres\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.709887 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.709790 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-proc\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.709887 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.709809 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-sys\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.811184 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.811153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-lib-modules\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.811331 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.811195 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44qc7\" (UniqueName: \"kubernetes.io/projected/36398fbd-60d9-4867-b360-5a17fc3aba2e-kube-api-access-44qc7\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.811331 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.811236 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-podres\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.811331 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.811260 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-proc\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.811331 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.811276 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-sys\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.811520 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.811348 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-sys\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.811520 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.811354 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-lib-modules\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.811520 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.811370 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-proc\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.811520 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.811356 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/36398fbd-60d9-4867-b360-5a17fc3aba2e-podres\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.819223 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.819195 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44qc7\" (UniqueName: \"kubernetes.io/projected/36398fbd-60d9-4867-b360-5a17fc3aba2e-kube-api-access-44qc7\") pod \"perf-node-gather-daemonset-2drnd\" (UID: \"36398fbd-60d9-4867-b360-5a17fc3aba2e\") " pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:08.906878 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:08.906843 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:09.046723 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:09.046648 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd"] Jul 17 19:34:09.049468 ip-10-0-132-220 kubenswrapper[2563]: W0717 19:34:09.049427 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod36398fbd_60d9_4867_b360_5a17fc3aba2e.slice/crio-1c3ae59f6b5b2f1c4719f39bbb7771a3e65da50137c5f5ce9c134d9229a51b5b WatchSource:0}: Error finding container 1c3ae59f6b5b2f1c4719f39bbb7771a3e65da50137c5f5ce9c134d9229a51b5b: Status 404 returned error can't find the container with id 1c3ae59f6b5b2f1c4719f39bbb7771a3e65da50137c5f5ce9c134d9229a51b5b Jul 17 19:34:10.042280 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:10.042242 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" event={"ID":"36398fbd-60d9-4867-b360-5a17fc3aba2e","Type":"ContainerStarted","Data":"c50e2a37348d4cf00346ed47f5f24139a9a40b3a98bf5bb42f984c02c5d7a0a6"} Jul 17 19:34:10.042799 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:10.042307 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" event={"ID":"36398fbd-60d9-4867-b360-5a17fc3aba2e","Type":"ContainerStarted","Data":"1c3ae59f6b5b2f1c4719f39bbb7771a3e65da50137c5f5ce9c134d9229a51b5b"} Jul 17 19:34:10.042799 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:10.042336 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:10.057321 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:10.057265 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" podStartSLOduration=2.057251256 podStartE2EDuration="2.057251256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 19:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 19:34:10.056339273 +0000 UTC m=+3666.959913174" watchObservedRunningTime="2026-07-17 19:34:10.057251256 +0000 UTC m=+3666.960825131" Jul 17 19:34:10.442508 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:10.442472 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-d2m7t_9d96a286-224f-4ff5-a5d1-79afd0d51a22/volume-data-source-validator/0.log" Jul 17 19:34:11.566029 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:11.565991 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x2dl7_8a782ebc-a204-4a31-a122-bd141b81b66d/dns/0.log" Jul 17 19:34:11.588231 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:11.588208 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x2dl7_8a782ebc-a204-4a31-a122-bd141b81b66d/kube-rbac-proxy/0.log" Jul 17 19:34:11.658916 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:11.658890 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tkrs6_f163c09b-fe4d-48de-9c56-7c8bbe212bc1/dns-node-resolver/0.log" Jul 17 19:34:12.282436 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:12.282403 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qnnsp_d2ca6a7a-5112-46a9-99fc-7c54e3ca7c8c/node-ca/0.log" Jul 17 19:34:13.259505 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:13.259476 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf484jg9_5b252de0-39c6-4fd8-9bf6-a9cee59a6c33/istio-proxy/0.log" Jul 17 19:34:13.400229 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:13.400200 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864488b8fd-6g5lc_05328159-9607-4cc5-898b-25388d78633b/kube-auth-proxy/0.log" Jul 17 19:34:13.456252 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:13.456219 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rvs2d_27310e84-6e5b-42d5-ba9a-ea4920dc6071/istio-proxy/0.log" Jul 17 19:34:13.515656 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:13.515566 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rvs2d_27310e84-6e5b-42d5-ba9a-ea4920dc6071/istio-proxy/1.log" Jul 17 19:34:13.555378 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:13.555344 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-srdlz_962e09eb-d2ca-41e4-ae57-16cfb445573c/payload-pre-processing/0.log" Jul 17 19:34:13.594305 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:13.594272 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-dc72j_8e68af24-81ea-4746-9d79-3ad158296036/payload-processing/0.log" Jul 17 19:34:13.616246 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:13.616215 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57c488db8b-cxf75_1c73c210-fc62-4860-93e4-c352a8021182/router/0.log" Jul 17 19:34:14.333419 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:14.333382 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kxtqg_9ec210e0-c89d-4835-b168-f3708c29d87f/serve-healthcheck-canary/0.log" Jul 17 19:34:14.939068 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:14.939018 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gn8pw_9dbad995-ab40-421a-bb52-4a6723f31ffd/kube-rbac-proxy/0.log" Jul 17 19:34:14.960954 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:14.960915 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gn8pw_9dbad995-ab40-421a-bb52-4a6723f31ffd/exporter/0.log" Jul 17 19:34:14.983374 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:14.983347 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gn8pw_9dbad995-ab40-421a-bb52-4a6723f31ffd/extractor/0.log" Jul 17 19:34:16.058718 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:16.058691 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dtqk6/perf-node-gather-daemonset-2drnd" Jul 17 19:34:17.266130 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:17.266104 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5c8qs_0db86b23-a5c6-4919-b36b-d05d33d44e6c/manager/0.log" Jul 17 19:34:17.511265 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:17.511177 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7bqrm_7a768798-edd2-49f7-81e6-8eb5d880c6f2/manager/0.log" Jul 17 19:34:17.672003 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:17.671977 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zqxbn_1d960e12-58ba-4994-b607-2e80aff93084/server/0.log" Jul 17 19:34:17.789730 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:17.789634 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-6prfl_1aa7760b-8a78-4310-b9ea-3482fcc94ba8/manager/0.log" Jul 17 19:34:17.876692 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:17.876653 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-qdzll_e4467a8e-8f06-4c08-890d-514366edb029/manager/0.log" Jul 17 19:34:17.901085 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:17.901055 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57479f7bfb-qdzll_e4467a8e-8f06-4c08-890d-514366edb029/copy-manifests/0.log" Jul 17 19:34:19.080184 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:19.080150 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf484jg9_5b252de0-39c6-4fd8-9bf6-a9cee59a6c33/istio-proxy/0.log" Jul 17 19:34:19.206812 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:19.206785 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864488b8fd-6g5lc_05328159-9607-4cc5-898b-25388d78633b/kube-auth-proxy/0.log" Jul 17 19:34:19.248807 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:19.248777 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rvs2d_27310e84-6e5b-42d5-ba9a-ea4920dc6071/istio-proxy/0.log" Jul 17 19:34:19.311016 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:19.310979 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rvs2d_27310e84-6e5b-42d5-ba9a-ea4920dc6071/istio-proxy/1.log" Jul 17 19:34:19.338362 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:19.338233 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-srdlz_962e09eb-d2ca-41e4-ae57-16cfb445573c/payload-pre-processing/0.log" Jul 17 19:34:19.367313 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:19.367277 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-dc72j_8e68af24-81ea-4746-9d79-3ad158296036/payload-processing/0.log" Jul 17 19:34:19.379533 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:19.379494 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57c488db8b-cxf75_1c73c210-fc62-4860-93e4-c352a8021182/router/0.log" Jul 17 19:34:19.403849 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:19.403816 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-85mpm_00eed65e-1fe2-441c-839b-075c6ccec781/openshift-lws-operator/0.log" Jul 17 19:34:20.568340 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:20.568307 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf484jg9_5b252de0-39c6-4fd8-9bf6-a9cee59a6c33/istio-proxy/0.log" Jul 17 19:34:20.688299 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:20.688268 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-864488b8fd-6g5lc_05328159-9607-4cc5-898b-25388d78633b/kube-auth-proxy/0.log" Jul 17 19:34:20.733094 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:20.733060 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rvs2d_27310e84-6e5b-42d5-ba9a-ea4920dc6071/istio-proxy/0.log" Jul 17 19:34:20.792043 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:20.791951 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-rvs2d_27310e84-6e5b-42d5-ba9a-ea4920dc6071/istio-proxy/1.log" Jul 17 19:34:20.836078 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:20.836048 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-srdlz_962e09eb-d2ca-41e4-ae57-16cfb445573c/payload-pre-processing/0.log" Jul 17 19:34:20.867083 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:20.867052 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-dc72j_8e68af24-81ea-4746-9d79-3ad158296036/payload-processing/0.log" Jul 17 19:34:20.881956 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:20.881921 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57c488db8b-cxf75_1c73c210-fc62-4860-93e4-c352a8021182/router/0.log" Jul 17 19:34:24.880229 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:24.880190 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-mt27w_3ff9293e-8e04-4639-b2e8-4e806d121427/kube-storage-version-migrator-operator/1.log" Jul 17 19:34:24.882053 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:24.882019 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-mt27w_3ff9293e-8e04-4639-b2e8-4e806d121427/kube-storage-version-migrator-operator/0.log" Jul 17 19:34:25.850666 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:25.850626 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4p2c6_031fe748-5d75-4fc9-a66a-de08e5d2b9d2/kube-multus/0.log" Jul 17 19:34:25.930304 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:25.930268 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kmz6_f3217f73-aa32-4cf7-b9ea-771cab429a4f/kube-multus-additional-cni-plugins/0.log" Jul 17 19:34:25.951517 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:25.951482 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kmz6_f3217f73-aa32-4cf7-b9ea-771cab429a4f/egress-router-binary-copy/0.log" Jul 17 19:34:25.971344 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:25.971319 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kmz6_f3217f73-aa32-4cf7-b9ea-771cab429a4f/cni-plugins/0.log" Jul 17 19:34:25.991727 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:25.991702 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kmz6_f3217f73-aa32-4cf7-b9ea-771cab429a4f/bond-cni-plugin/0.log" Jul 17 19:34:26.011169 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:26.011140 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kmz6_f3217f73-aa32-4cf7-b9ea-771cab429a4f/routeoverride-cni/0.log" Jul 17 19:34:26.032205 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:26.032138 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kmz6_f3217f73-aa32-4cf7-b9ea-771cab429a4f/whereabouts-cni-bincopy/0.log" Jul 17 19:34:26.053709 ip-10-0-132-220 kubenswrapper[2563]: I0717 19:34:26.053669 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5kmz6_f3217f73-aa32-4cf7-b9ea-771cab429a4f/whereabouts-cni/0.log"