Jul 08 19:11:22.571632 ip-10-0-128-188 systemd[1]: Starting Kubernetes Kubelet... Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:22.999021 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.002990 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003002 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003005 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003009 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003012 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003015 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003018 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:11:23.051286 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003021 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003025 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003028 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003030 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003033 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003036 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003038 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003041 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003044 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003047 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003049 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003052 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003055 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003058 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003066 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003071 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003075 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003078 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003080 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:11:23.053259 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003083 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003086 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003089 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003091 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003094 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003096 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003099 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003102 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003104 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003107 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003109 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003112 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003114 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003116 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003119 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003122 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003125 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003127 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003130 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003133 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:11:23.055453 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003135 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003138 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003141 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003143 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003146 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003148 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003151 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003153 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003156 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003159 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003162 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003165 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003167 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003169 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003172 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003174 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003177 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003182 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003185 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003188 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:11:23.071143 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003191 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:11:23.071465 ip-10-0-128-188 systemd[1]: Started Kubernetes Kubelet. Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003193 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003196 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003200 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003202 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003205 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003208 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003211 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003214 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003216 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003219 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003222 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003225 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003231 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003234 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003236 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003239 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003241 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003244 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003246 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:11:23.074653 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003249 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003252 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003254 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003257 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003260 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003263 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003265 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003268 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003271 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003273 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003276 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003280 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003283 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003977 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003983 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003986 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003989 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003992 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003994 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:11:23.077789 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.003997 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004000 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004002 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004004 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004007 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004010 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004013 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004015 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004019 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004022 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004024 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004027 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004029 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004032 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004035 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004037 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004040 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004042 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004045 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004047 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:11:23.093151 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004051 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004053 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004056 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004059 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004061 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004064 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004067 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004070 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004072 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004075 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004078 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004080 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004083 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004086 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004088 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004091 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004093 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004096 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004098 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:11:23.095192 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004101 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004103 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004106 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004109 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004113 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004117 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004121 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004123 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004126 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004129 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004131 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004134 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004137 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004140 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004142 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004145 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004147 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004150 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004152 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004155 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:11:23.097587 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004158 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004160 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004163 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004165 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004168 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004170 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004173 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004176 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004178 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004181 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004183 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004189 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004192 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004195 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004197 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004202 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004205 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004209 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004213 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:11:23.111076 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004216 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004219 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004222 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004224 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004227 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004231 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004234 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004237 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004240 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004243 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004246 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004248 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004252 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004254 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004258 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004642 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004648 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004652 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004656 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:11:23.113727 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004659 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004662 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004665 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004668 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004671 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004676 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004678 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004681 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004684 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004687 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004689 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004692 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004695 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004697 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004700 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004702 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004705 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004707 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004710 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004713 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:11:23.115719 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004716 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004718 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004721 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004723 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004726 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004729 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004731 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004734 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004737 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004740 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004742 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004745 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004747 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004750 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004754 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004757 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004760 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004764 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004766 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004769 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:11:23.119087 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004772 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004774 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004777 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004780 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004782 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004785 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004787 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004790 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004792 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004795 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004797 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004800 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004802 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004805 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004808 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004810 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004813 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004816 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004819 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004823 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:11:23.121235 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004825 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004828 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004831 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004834 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004837 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004839 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004843 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004846 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004849 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004853 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004856 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004858 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004861 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004863 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004866 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004869 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004872 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004875 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004877 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004880 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:11:23.123803 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004883 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004885 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004888 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004890 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004893 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004895 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004898 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004900 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004903 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004905 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004908 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004911 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004913 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004917 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.004920 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.004989 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.004995 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005001 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005007 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005010 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005014 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 19:11:23.124477 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005019 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005023 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005026 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005029 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005032 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005036 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005039 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005042 2576 flags.go:64] FLAG: --cgroup-root="" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005045 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005048 2576 flags.go:64] FLAG: --client-ca-file="" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005051 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005054 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005058 2576 flags.go:64] FLAG: --cluster-domain="" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005060 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005063 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005066 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005069 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005073 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005076 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005079 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005082 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005085 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005088 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005091 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 19:11:23.125142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005094 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005097 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005101 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005104 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005108 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005112 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005115 2576 flags.go:64] FLAG: --enable-server="true" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005118 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005123 2576 flags.go:64] FLAG: --event-burst="100" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005126 2576 flags.go:64] FLAG: --event-qps="50" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005129 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005132 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005135 2576 flags.go:64] FLAG: --eviction-hard="" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005139 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005142 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005145 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005150 2576 flags.go:64] FLAG: --eviction-soft="" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005153 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005156 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005158 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005161 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005164 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005167 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005170 2576 flags.go:64] FLAG: --feature-gates="" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005173 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 19:11:23.125817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005176 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005179 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005182 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005186 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005188 2576 flags.go:64] FLAG: --help="false" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005191 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005194 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005197 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005200 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005203 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005207 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005210 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005213 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005216 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005219 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005223 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005226 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005229 2576 flags.go:64] FLAG: --kube-reserved="" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005232 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005235 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005238 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005241 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005243 2576 flags.go:64] FLAG: --lock-file="" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005246 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 19:11:23.126588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005249 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005252 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005257 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005260 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005263 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005266 2576 flags.go:64] FLAG: --logging-format="text" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005269 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005272 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005275 2576 flags.go:64] FLAG: --manifest-url="" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005278 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005282 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005285 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005289 2576 flags.go:64] FLAG: --max-pods="110" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005292 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005294 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005297 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005300 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005304 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005306 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005309 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005317 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005320 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005323 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 19:11:23.127544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005328 2576 flags.go:64] FLAG: --pod-cidr="" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005331 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005337 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005355 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005359 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005362 2576 flags.go:64] FLAG: --port="10250" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005365 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005368 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e9b2bbff8d1b6744" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005372 2576 flags.go:64] FLAG: --qos-reserved="" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005375 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005378 2576 flags.go:64] FLAG: --register-node="true" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005381 2576 flags.go:64] FLAG: --register-with-taints="" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005385 2576 flags.go:64] FLAG: --registry-burst="10" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005388 2576 flags.go:64] FLAG: --registry-qps="5" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005391 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005394 2576 flags.go:64] FLAG: --reserved-memory="" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005397 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005400 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005403 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005406 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005409 2576 flags.go:64] FLAG: --runonce="false" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005412 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005415 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005418 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005420 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 19:11:23.128300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005423 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005426 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005429 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005432 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005436 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005439 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005442 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005445 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005449 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005452 2576 flags.go:64] FLAG: --system-cgroups="" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005454 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005460 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005463 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005466 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005470 2576 flags.go:64] FLAG: --tls-min-version="" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005472 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005475 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005478 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005481 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005484 2576 flags.go:64] FLAG: --v="2" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005488 2576 flags.go:64] FLAG: --version="false" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005497 2576 flags.go:64] FLAG: --vmodule="" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005504 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005507 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005617 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:11:23.287975 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005620 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005623 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005626 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005629 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005632 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005635 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005638 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005640 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005643 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005646 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005648 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005652 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005655 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005658 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005660 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005664 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005667 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005669 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005672 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005675 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:11:23.325142 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005677 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005680 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005682 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005685 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005687 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005690 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005693 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005695 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005698 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005700 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005703 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005705 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005708 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005711 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005713 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005715 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005718 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005720 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005723 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005725 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:11:23.327305 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005728 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005730 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005733 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005737 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005740 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005742 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005745 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005749 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005753 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005757 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005759 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005762 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005765 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005768 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005770 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005773 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005775 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005778 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005780 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:11:23.328838 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005783 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005786 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005788 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005791 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005794 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005796 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005799 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005801 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005804 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005807 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005809 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005811 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005814 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005816 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005819 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005822 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005825 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005828 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005831 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:11:23.329892 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005835 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005839 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005842 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005845 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005849 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005853 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005856 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005859 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005862 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005864 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005867 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005870 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005872 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005875 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005878 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005880 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005883 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005885 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005888 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:11:23.330517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.005890 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.005899 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.012973 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.012991 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.013009 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.013015 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013076 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013081 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013084 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013088 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013090 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013093 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013096 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013098 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:11:23.331146 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013101 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013104 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013106 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013108 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013111 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013115 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013119 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013122 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013124 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013127 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013130 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013132 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013135 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013138 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013140 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013143 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013145 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013148 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013151 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:11:23.331767 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013153 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013158 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013167 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013169 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013172 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013175 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013177 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013180 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013183 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013185 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013188 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013190 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013193 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013195 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013198 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013200 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013203 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013205 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013207 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013210 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:11:23.332968 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013212 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013215 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013217 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013220 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013222 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013225 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013229 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013232 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013235 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013238 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013240 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013243 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013246 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013250 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013252 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013262 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013265 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013268 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013271 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:11:23.333592 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013274 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013276 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013279 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013282 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013284 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013287 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013289 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013292 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013294 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013297 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013300 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013302 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013304 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013307 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013309 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013312 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013315 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013317 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013320 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013322 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:11:23.334121 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013325 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013327 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013330 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013333 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013336 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013338 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013357 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013360 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013362 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013371 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013374 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013377 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013379 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.013384 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013529 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:11:23.334729 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013534 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013537 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013540 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013543 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013546 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013548 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013551 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013554 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013556 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013559 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013561 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013564 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013566 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013569 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013572 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013576 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013579 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013582 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013585 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013588 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:11:23.335148 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013590 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013593 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013596 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013599 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013602 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013605 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013608 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013616 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013619 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013621 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013624 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013627 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013631 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013634 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013637 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013639 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013642 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013644 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013647 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013649 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:11:23.335749 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013651 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013654 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013656 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013659 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013661 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013664 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013666 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013669 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013671 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013674 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013676 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013678 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013681 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013686 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013688 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013691 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013695 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013697 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013700 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:11:23.336287 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013703 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013712 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013714 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013717 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013720 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013722 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013726 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013729 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013732 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013735 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013738 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013741 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013744 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013746 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013749 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013752 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013754 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013757 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013760 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:11:23.336854 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013763 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013765 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013768 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013770 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013772 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013775 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013778 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013782 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013784 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013787 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013791 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013793 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013796 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013798 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013801 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013809 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013812 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013814 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013817 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:11:23.337365 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:23.013819 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.013824 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.013939 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.015952 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.017476 2576 server.go:1019] "Starting client certificate rotation" Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.017579 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.017633 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.021455 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.022796 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.034315 2576 log.go:25] "Validated CRI v1 runtime API" Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.037469 2576 log.go:25] "Validated CRI v1 image API" Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.038194 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 19:11:23.337928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.040645 2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ac408da9-741c-4d30-8fb2-631259e07e81:/dev/nvme0n1p3 b4c9d58e-bce9-42fd-b434-33adcf8a046d:/dev/nvme0n1p4] Jul 08 19:11:23.338291 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.040661 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 19:11:23.338449 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.046434 2576 manager.go:216] Machine: {Timestamp:2026-07-08 19:11:23.044365906 +0000 UTC m=+0.346670063 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102248 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2adb33f7e8fd3c0cea70938c5f7319 SystemUUID:ec2adb33-f7e8-fd3c-0cea-70938c5f7319 BootID:e5be0e3e-bfd3-46be-a8e5-8c16dc91a260 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b9:5b:db:1e:4b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b9:5b:db:1e:4b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:97:f4:84:5e:f2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 19:11:23.338449 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.046540 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 19:11:23.338449 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.046619 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 19:11:23.338449 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.048153 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 19:11:23.338449 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.048181 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-188.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 19:11:23.338449 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.049063 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 19:11:23.338449 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.049075 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 19:11:23.338449 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.049082 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.050579 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.050728 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.053227 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.056547 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.056661 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.056688 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.056697 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.056713 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.056722 2576 kubelet.go:398] "Adding apiserver pod source" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.056736 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.057145 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.057168 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.061860 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.063864 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.063892 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065076 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065089 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065095 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065101 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065110 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065116 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065125 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065130 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065140 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065145 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065154 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065159 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065980 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.065987 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.067409 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.067549 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.069532 2576 server.go:1268] "Started kubelet" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.069676 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.069761 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.069792 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.072323 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.074879 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.076621 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.080422 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 19:11:23.339037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.080439 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.080515 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-188.ec2.internal\" not found" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081053 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081070 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081069 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081163 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081170 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081233 2576 factory.go:55] Registering systemd factory Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081248 2576 factory.go:223] Registration of the systemd container factory successfully Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081370 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-188.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081537 2576 factory.go:153] Registering CRI-O factory Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081550 2576 factory.go:223] Registration of the crio container factory successfully Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081601 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081618 2576 factory.go:103] Registering Raw factory Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.081630 2576 manager.go:1196] Started watching for new ooms in manager Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.082001 2576 manager.go:349] Starting recovery of all containers Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.081050 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-188.ec2.internal.18c066414bc9c194 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-188.ec2.internal,UID:ip-10-0-128-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-188.ec2.internal,},FirstTimestamp:2026-07-08 19:11:23.0691373 +0000 UTC m=+0.371441458,LastTimestamp:2026-07-08 19:11:23.0691373 +0000 UTC m=+0.371441458,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-188.ec2.internal,}" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.089407 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.089674 2576 manager.go:354] Recovery completed Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.093161 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.093268 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.095012 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.096991 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.097025 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.097035 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.097585 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.097597 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.097598 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.097642 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.098069 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tj2nr" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.098319 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:11:23.340129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.098361 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.098375 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.098869 2576 policy_none.go:49] "None policy: Start" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.098886 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.098900 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.100092 2576 policy_none.go:47] "Start" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.105944 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tj2nr" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.107855 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-188.ec2.internal.18c066414d73100c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-188.ec2.internal,UID:ip-10-0-128-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-188.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-188.ec2.internal,},FirstTimestamp:2026-07-08 19:11:23.097010188 +0000 UTC m=+0.399314345,LastTimestamp:2026-07-08 19:11:23.097010188 +0000 UTC m=+0.399314345,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-188.ec2.internal,}" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.140906 2576 manager.go:337] "Starting Device Plugin manager" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.141061 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.141073 2576 server.go:87] "Starting device plugin registration server" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.141261 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.141270 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.141384 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.141462 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.141472 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.141883 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.141938 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-188.ec2.internal\" not found" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.242167 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.243112 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.243148 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.243162 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.243187 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.250867 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.250888 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-188.ec2.internal\": node \"ip-10-0-128-188.ec2.internal\" not found" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.261368 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.261411 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.261432 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.261463 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.263897 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.276433 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-188.ec2.internal\" not found" Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.364718 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal"] Jul 08 19:11:23.438078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.364798 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.365680 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.365711 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.365723 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.367114 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.367251 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.367277 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.367812 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.367840 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.367855 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.367857 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.367875 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.367883 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.369193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.369214 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.369822 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.369838 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.369853 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.377236 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-188.ec2.internal\" not found" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.383566 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5117d2d045a1c775b90c668c570fae47-config\") pod \"kube-apiserver-proxy-ip-10-0-128-188.ec2.internal\" (UID: \"5117d2d045a1c775b90c668c570fae47\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.383587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cfe8542469e4bef4aeca33118acda14c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal\" (UID: \"cfe8542469e4bef4aeca33118acda14c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.383603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfe8542469e4bef4aeca33118acda14c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal\" (UID: \"cfe8542469e4bef4aeca33118acda14c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.408597 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-188.ec2.internal\" not found" node="ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.439292 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.413431 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-188.ec2.internal\" not found" node="ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.540840 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.477745 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-188.ec2.internal\" not found" Jul 08 19:11:23.540840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.484418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5117d2d045a1c775b90c668c570fae47-config\") pod \"kube-apiserver-proxy-ip-10-0-128-188.ec2.internal\" (UID: \"5117d2d045a1c775b90c668c570fae47\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.540840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.484452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cfe8542469e4bef4aeca33118acda14c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal\" (UID: \"cfe8542469e4bef4aeca33118acda14c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.540840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.484476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfe8542469e4bef4aeca33118acda14c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal\" (UID: \"cfe8542469e4bef4aeca33118acda14c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.540840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.484522 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5117d2d045a1c775b90c668c570fae47-config\") pod \"kube-apiserver-proxy-ip-10-0-128-188.ec2.internal\" (UID: \"5117d2d045a1c775b90c668c570fae47\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.540840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.484529 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfe8542469e4bef4aeca33118acda14c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal\" (UID: \"cfe8542469e4bef4aeca33118acda14c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.540840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.484529 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cfe8542469e4bef4aeca33118acda14c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal\" (UID: \"cfe8542469e4bef4aeca33118acda14c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.597804 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.578805 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-188.ec2.internal\" not found" Jul 08 19:11:23.722363 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.679376 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-188.ec2.internal\" not found" Jul 08 19:11:23.722363 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.711831 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.722363 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.716745 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal" Jul 08 19:11:23.793323 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.780501 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-188.ec2.internal\" not found" Jul 08 19:11:23.917198 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:23.881059 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-188.ec2.internal\" not found" Jul 08 19:11:23.917198 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.896318 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:11:24.035308 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.980653 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" Jul 08 19:11:24.035308 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.995090 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 19:11:24.035308 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:23.998471 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal" Jul 08 19:11:24.035308 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.008222 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 19:11:24.035308 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.016337 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 19:11:24.035308 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.016463 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.057734 2576 apiserver.go:52] "Watching apiserver" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.070171 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.071197 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-6qs6k","kube-system/konnectivity-agent-zqct5","openshift-cluster-node-tuning-operator/tuned-d48zc","openshift-dns/node-resolver-zzbhq","openshift-image-registry/node-ca-lldql","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal","openshift-multus/multus-j5pbq","openshift-network-diagnostics/network-check-target-b27qk","kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd","openshift-multus/multus-additional-cni-plugins-26h9r","openshift-multus/network-metrics-daemon-nfnnj","openshift-network-operator/iptables-alerter-crrcv","openshift-ovn-kubernetes/ovnkube-node-hr5zg"] Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.072691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.072748 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.073760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.075025 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.076470 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.077663 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.077878 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078054 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078065 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078158 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078168 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d6bj5\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078237 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078263 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078327 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078354 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078449 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-c69kn\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078699 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.078963 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.079020 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.079082 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.079458 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.079487 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l9x4w\"" Jul 08 19:11:24.159231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.080282 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.080342 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.080567 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.080758 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.080762 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.080989 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.081717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.082311 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dl66h\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.083363 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.083471 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nrgpm\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.083620 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.083936 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.084214 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.084877 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.084895 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.085167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.085260 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.085347 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xgz9h\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.085408 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.086310 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.086413 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s6xhd\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.086492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.086752 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.088076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089074 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089212 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2mxsm\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089315 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-systemd-units\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.160867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-log-socket\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089620 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-ovn-node-metrics-cert\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-system-cni-dir\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-run-k8s-cni-cncf-io\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/043c26a6-ec7b-4054-9107-e51a93ad9536-iptables-alerter-script\") pod \"iptables-alerter-crrcv\" (UID: \"043c26a6-ec7b-4054-9107-e51a93ad9536\") " pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089817 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-device-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-cni-netd\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-ovnkube-config\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-multus-conf-dir\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b982417-f3a9-43ff-9a76-920bd553524e-kubelet-config\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.089968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndpn2\" (UniqueName: \"kubernetes.io/projected/043c26a6-ec7b-4054-9107-e51a93ad9536-kube-api-access-ndpn2\") pod \"iptables-alerter-crrcv\" (UID: \"043c26a6-ec7b-4054-9107-e51a93ad9536\") " pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-kubelet\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.161833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-run-openvswitch\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-multus-socket-dir-parent\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-var-lib-kubelet\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-node-log\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d385dcd3-5579-42a7-af3d-c9f272c9a568-host\") pod \"node-ca-lldql\" (UID: \"d385dcd3-5579-42a7-af3d-c9f272c9a568\") " pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-slash\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d385dcd3-5579-42a7-af3d-c9f272c9a568-serviceca\") pod \"node-ca-lldql\" (UID: \"d385dcd3-5579-42a7-af3d-c9f272c9a568\") " pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-registration-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090308 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-etc-openvswitch\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-run-ovn\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zck88\" (UniqueName: \"kubernetes.io/projected/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-kube-api-access-zck88\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-run-netns\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/043c26a6-ec7b-4054-9107-e51a93ad9536-host-slash\") pod \"iptables-alerter-crrcv\" (UID: \"043c26a6-ec7b-4054-9107-e51a93ad9536\") " pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.163768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-os-release\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-var-lib-cni-multus\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ssw8\" (UniqueName: \"kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8\") pod \"network-check-target-b27qk\" (UID: \"429a71ea-4e0e-4573-8394-bcbe65957d13\") " pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-var-lib-openvswitch\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-hostroot\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090862 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1627836b-2975-43af-b809-0476207e62b2-multus-daemon-config\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090907 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvngn\" (UniqueName: \"kubernetes.io/projected/1627836b-2975-43af-b809-0476207e62b2-kube-api-access-jvngn\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.090981 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbqxv\" (UniqueName: \"kubernetes.io/projected/d777b857-7691-4edc-b186-2170dde274d3-kube-api-access-tbqxv\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-cni-bin\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091044 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cxh7t\"" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091046 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-cnibin\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:24.164730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091162 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca-hosts-file\") pod \"node-resolver-zzbhq\" (UID: \"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca\") " pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091180 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-run-systemd\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-ovnkube-script-lib\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-etc-kubernetes\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b982417-f3a9-43ff-9a76-920bd553524e-dbus\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5qx\" (UniqueName: \"kubernetes.io/projected/1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca-kube-api-access-sm5qx\") pod \"node-resolver-zzbhq\" (UID: \"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca\") " pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091446 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q79h\" (UniqueName: \"kubernetes.io/projected/d385dcd3-5579-42a7-af3d-c9f272c9a568-kube-api-access-6q79h\") pod \"node-ca-lldql\" (UID: \"d385dcd3-5579-42a7-af3d-c9f272c9a568\") " pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-run-netns\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-env-overrides\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-multus-cni-dir\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-run-multus-certs\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca-tmp-dir\") pod \"node-resolver-zzbhq\" (UID: \"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca\") " pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-var-lib-cni-bin\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.167120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091703 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-socket-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.168093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.091769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-sys-fs\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.168093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.092139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1627836b-2975-43af-b809-0476207e62b2-cni-binary-copy\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.168093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.099920 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 19:11:24.168093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.108766 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 19:06:23 +0000 UTC" deadline="2027-12-25 22:32:18.292262358 +0000 UTC" Jul 08 19:11:24.168093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.108797 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12843h20m54.183467436s" Jul 08 19:11:24.168093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.130943 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k2lsp" Jul 08 19:11:24.168093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.140374 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k2lsp" Jul 08 19:11:24.168093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.154969 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:11:24.184958 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.182331 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-run\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-tuned\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192574 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-cni-netd\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-ovnkube-config\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-multus-conf-dir\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b982417-f3a9-43ff-9a76-920bd553524e-kubelet-config\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192696 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-multus-conf-dir\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpn2\" (UniqueName: \"kubernetes.io/projected/043c26a6-ec7b-4054-9107-e51a93ad9536-kube-api-access-ndpn2\") pod \"iptables-alerter-crrcv\" (UID: \"043c26a6-ec7b-4054-9107-e51a93ad9536\") " pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-cni-netd\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192732 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b982417-f3a9-43ff-9a76-920bd553524e-kubelet-config\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-kubelet\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-kubernetes\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkd9z\" (UniqueName: \"kubernetes.io/projected/d7766fe7-0b38-4992-91e4-4079502ca421-kube-api-access-qkd9z\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.308284 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192839 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-kubelet\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-run-openvswitch\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192891 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-multus-socket-dir-parent\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-var-lib-kubelet\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192944 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-run-openvswitch\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-systemd\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192984 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-run-ovn-kubernetes\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.192994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-sys\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193023 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-var-lib-kubelet\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-lib-modules\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193061 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-multus-socket-dir-parent\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59a6b211-27a3-4086-914f-cfb2a1418f9b-konnectivity-ca\") pod \"konnectivity-agent-zqct5\" (UID: \"59a6b211-27a3-4086-914f-cfb2a1418f9b\") " pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-node-log\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d385dcd3-5579-42a7-af3d-c9f272c9a568-host\") pod \"node-ca-lldql\" (UID: \"d385dcd3-5579-42a7-af3d-c9f272c9a568\") " pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.309227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193124 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-node-log\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7766fe7-0b38-4992-91e4-4079502ca421-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193160 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d385dcd3-5579-42a7-af3d-c9f272c9a568-host\") pod \"node-ca-lldql\" (UID: \"d385dcd3-5579-42a7-af3d-c9f272c9a568\") " pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7766fe7-0b38-4992-91e4-4079502ca421-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-slash\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d385dcd3-5579-42a7-af3d-c9f272c9a568-serviceca\") pod \"node-ca-lldql\" (UID: \"d385dcd3-5579-42a7-af3d-c9f272c9a568\") " pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-registration-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193257 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-slash\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193277 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-registration-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-sysconfig\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-sysctl-conf\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193318 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-tuning-conf-dir\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-etc-openvswitch\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-run-ovn\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.311625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zck88\" (UniqueName: \"kubernetes.io/projected/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-kube-api-access-zck88\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193393 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-etc-openvswitch\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-run-netns\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/043c26a6-ec7b-4054-9107-e51a93ad9536-host-slash\") pod \"iptables-alerter-crrcv\" (UID: \"043c26a6-ec7b-4054-9107-e51a93ad9536\") " pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-run-ovn\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193454 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-run-netns\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-var-lib-kubelet\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193488 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/043c26a6-ec7b-4054-9107-e51a93ad9536-host-slash\") pod \"iptables-alerter-crrcv\" (UID: \"043c26a6-ec7b-4054-9107-e51a93ad9536\") " pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193497 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-os-release\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193528 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-var-lib-cni-multus\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssw8\" (UniqueName: \"kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8\") pod \"network-check-target-b27qk\" (UID: \"429a71ea-4e0e-4573-8394-bcbe65957d13\") " pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-var-lib-cni-multus\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e94d3f2a-be4c-48b7-b12b-05ac1927e767-tmp\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193579 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-os-release\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-var-lib-openvswitch\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.312152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-system-cni-dir\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-hostroot\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193671 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-var-lib-openvswitch\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1627836b-2975-43af-b809-0476207e62b2-multus-daemon-config\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193682 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvngn\" (UniqueName: \"kubernetes.io/projected/1627836b-2975-43af-b809-0476207e62b2-kube-api-access-jvngn\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193724 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-hostroot\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqxv\" (UniqueName: \"kubernetes.io/projected/d777b857-7691-4edc-b186-2170dde274d3-kube-api-access-tbqxv\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-sysctl-d\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-cni-bin\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-cnibin\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193864 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-etc-selinux\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.314094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca-hosts-file\") pod \"node-resolver-zzbhq\" (UID: \"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca\") " pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193919 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-cnibin\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193916 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-cni-bin\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgnw\" (UniqueName: \"kubernetes.io/projected/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-kube-api-access-kjgnw\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-host\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.193968 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca-hosts-file\") pod \"node-resolver-zzbhq\" (UID: \"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca\") " pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59a6b211-27a3-4086-914f-cfb2a1418f9b-agent-certs\") pod \"konnectivity-agent-zqct5\" (UID: \"59a6b211-27a3-4086-914f-cfb2a1418f9b\") " pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7766fe7-0b38-4992-91e4-4079502ca421-cni-binary-copy\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.193974 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-run-systemd\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-ovnkube-script-lib\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194102 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-run-systemd\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.194123 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret podName:5b982417-f3a9-43ff-9a76-920bd553524e nodeName:}" failed. No retries permitted until 2026-07-08 19:11:24.694081424 +0000 UTC m=+1.996385584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret") pod "global-pull-secret-syncer-6qs6k" (UID: "5b982417-f3a9-43ff-9a76-920bd553524e") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:24.315061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-etc-kubernetes\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194164 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-etc-kubernetes\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b982417-f3a9-43ff-9a76-920bd553524e-dbus\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5qx\" (UniqueName: \"kubernetes.io/projected/1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca-kube-api-access-sm5qx\") pod \"node-resolver-zzbhq\" (UID: \"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca\") " pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q79h\" (UniqueName: \"kubernetes.io/projected/d385dcd3-5579-42a7-af3d-c9f272c9a568-kube-api-access-6q79h\") pod \"node-ca-lldql\" (UID: \"d385dcd3-5579-42a7-af3d-c9f272c9a568\") " pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194276 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b982417-f3a9-43ff-9a76-920bd553524e-dbus\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-modprobe-d\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-run-netns\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194374 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-env-overrides\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-multus-cni-dir\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-run-multus-certs\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca-tmp-dir\") pod \"node-resolver-zzbhq\" (UID: \"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca\") " pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88fjj\" (UniqueName: \"kubernetes.io/projected/e94d3f2a-be4c-48b7-b12b-05ac1927e767-kube-api-access-88fjj\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.316035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-multus-cni-dir\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-cnibin\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194491 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-host-run-netns\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194501 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-run-multus-certs\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-var-lib-cni-bin\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-socket-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194556 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-var-lib-cni-bin\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-sys-fs\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-os-release\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194606 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-sys-fs\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1627836b-2975-43af-b809-0476207e62b2-cni-binary-copy\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-systemd-units\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-log-socket\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-ovn-node-metrics-cert\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.316960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194696 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-system-cni-dir\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-run-k8s-cni-cncf-io\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/043c26a6-ec7b-4054-9107-e51a93ad9536-iptables-alerter-script\") pod \"iptables-alerter-crrcv\" (UID: \"043c26a6-ec7b-4054-9107-e51a93ad9536\") " pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-device-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.194822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-device-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.195514 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-socket-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.195612 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-system-cni-dir\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.195657 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-systemd-units\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.195700 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-log-socket\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.195987 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-ovnkube-config\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.196009 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.196632 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-env-overrides\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.196760 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-ovnkube-script-lib\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.196770 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d385dcd3-5579-42a7-af3d-c9f272c9a568-serviceca\") pod \"node-ca-lldql\" (UID: \"d385dcd3-5579-42a7-af3d-c9f272c9a568\") " pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.318953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.196758 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1627836b-2975-43af-b809-0476207e62b2-host-run-k8s-cni-cncf-io\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.196810 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d777b857-7691-4edc-b186-2170dde274d3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.197020 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca-tmp-dir\") pod \"node-resolver-zzbhq\" (UID: \"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca\") " pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.197470 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/043c26a6-ec7b-4054-9107-e51a93ad9536-iptables-alerter-script\") pod \"iptables-alerter-crrcv\" (UID: \"043c26a6-ec7b-4054-9107-e51a93ad9536\") " pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.197483 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1627836b-2975-43af-b809-0476207e62b2-cni-binary-copy\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.197523 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1627836b-2975-43af-b809-0476207e62b2-multus-daemon-config\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.199188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-ovn-node-metrics-cert\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.203200 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.203220 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.203234 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6ssw8 for pod openshift-network-diagnostics/network-check-target-b27qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.203297 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8 podName:429a71ea-4e0e-4573-8394-bcbe65957d13 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:24.703280058 +0000 UTC m=+2.005584209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6ssw8" (UniqueName: "kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8") pod "network-check-target-b27qk" (UID: "429a71ea-4e0e-4573-8394-bcbe65957d13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.205315 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpn2\" (UniqueName: \"kubernetes.io/projected/043c26a6-ec7b-4054-9107-e51a93ad9536-kube-api-access-ndpn2\") pod \"iptables-alerter-crrcv\" (UID: \"043c26a6-ec7b-4054-9107-e51a93ad9536\") " pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.205502 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q79h\" (UniqueName: \"kubernetes.io/projected/d385dcd3-5579-42a7-af3d-c9f272c9a568-kube-api-access-6q79h\") pod \"node-ca-lldql\" (UID: \"d385dcd3-5579-42a7-af3d-c9f272c9a568\") " pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.205967 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqxv\" (UniqueName: \"kubernetes.io/projected/d777b857-7691-4edc-b186-2170dde274d3-kube-api-access-tbqxv\") pod \"aws-ebs-csi-driver-node-b94vd\" (UID: \"d777b857-7691-4edc-b186-2170dde274d3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.319831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.206022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5qx\" (UniqueName: \"kubernetes.io/projected/1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca-kube-api-access-sm5qx\") pod \"node-resolver-zzbhq\" (UID: \"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca\") " pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.206106 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zck88\" (UniqueName: \"kubernetes.io/projected/36b08b02-b9a1-4d19-836a-5cc004fb0c0d-kube-api-access-zck88\") pod \"ovnkube-node-hr5zg\" (UID: \"36b08b02-b9a1-4d19-836a-5cc004fb0c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.206342 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvngn\" (UniqueName: \"kubernetes.io/projected/1627836b-2975-43af-b809-0476207e62b2-kube-api-access-jvngn\") pod \"multus-j5pbq\" (UID: \"1627836b-2975-43af-b809-0476207e62b2\") " pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.295996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e94d3f2a-be4c-48b7-b12b-05ac1927e767-tmp\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-system-cni-dir\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-sysctl-d\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296092 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-system-cni-dir\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgnw\" (UniqueName: \"kubernetes.io/projected/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-kube-api-access-kjgnw\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-host\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59a6b211-27a3-4086-914f-cfb2a1418f9b-agent-certs\") pod \"konnectivity-agent-zqct5\" (UID: \"59a6b211-27a3-4086-914f-cfb2a1418f9b\") " pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7766fe7-0b38-4992-91e4-4079502ca421-cni-binary-copy\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296215 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-host\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296264 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-sysctl-d\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.322716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-modprobe-d\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88fjj\" (UniqueName: \"kubernetes.io/projected/e94d3f2a-be4c-48b7-b12b-05ac1927e767-kube-api-access-88fjj\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296332 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-modprobe-d\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-cnibin\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-os-release\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-run\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.296447 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-tuned\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-kubernetes\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkd9z\" (UniqueName: \"kubernetes.io/projected/d7766fe7-0b38-4992-91e4-4079502ca421-kube-api-access-qkd9z\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-systemd\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296594 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-run\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-sys\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-lib-modules\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59a6b211-27a3-4086-914f-cfb2a1418f9b-konnectivity-ca\") pod \"konnectivity-agent-zqct5\" (UID: \"59a6b211-27a3-4086-914f-cfb2a1418f9b\") " pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:24.323642 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296675 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-os-release\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7766fe7-0b38-4992-91e4-4079502ca421-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.296723 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs podName:8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:24.796701462 +0000 UTC m=+2.099005610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs") pod "network-metrics-daemon-nfnnj" (UID: "8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296638 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-cnibin\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296756 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7766fe7-0b38-4992-91e4-4079502ca421-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-sysconfig\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-sysctl-conf\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296825 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-lib-modules\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-tuning-conf-dir\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296894 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-kubernetes\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-var-lib-kubelet\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.296981 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7766fe7-0b38-4992-91e4-4079502ca421-tuning-conf-dir\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.297059 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-systemd\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.325995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.297059 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-sys\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.297121 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-var-lib-kubelet\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.297166 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-sysconfig\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.297283 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-sysctl-conf\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.297350 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/59a6b211-27a3-4086-914f-cfb2a1418f9b-konnectivity-ca\") pod \"konnectivity-agent-zqct5\" (UID: \"59a6b211-27a3-4086-914f-cfb2a1418f9b\") " pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.297654 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7766fe7-0b38-4992-91e4-4079502ca421-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.297697 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7766fe7-0b38-4992-91e4-4079502ca421-cni-binary-copy\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.297913 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7766fe7-0b38-4992-91e4-4079502ca421-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.298658 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e94d3f2a-be4c-48b7-b12b-05ac1927e767-etc-tuned\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.298776 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e94d3f2a-be4c-48b7-b12b-05ac1927e767-tmp\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.299099 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/59a6b211-27a3-4086-914f-cfb2a1418f9b-agent-certs\") pod \"konnectivity-agent-zqct5\" (UID: \"59a6b211-27a3-4086-914f-cfb2a1418f9b\") " pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.305578 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgnw\" (UniqueName: \"kubernetes.io/projected/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-kube-api-access-kjgnw\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.305586 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88fjj\" (UniqueName: \"kubernetes.io/projected/e94d3f2a-be4c-48b7-b12b-05ac1927e767-kube-api-access-88fjj\") pod \"tuned-d48zc\" (UID: \"e94d3f2a-be4c-48b7-b12b-05ac1927e767\") " pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.326485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.306305 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkd9z\" (UniqueName: \"kubernetes.io/projected/d7766fe7-0b38-4992-91e4-4079502ca421-kube-api-access-qkd9z\") pod \"multus-additional-cni-plugins-26h9r\" (UID: \"d7766fe7-0b38-4992-91e4-4079502ca421\") " pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.494568 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.484637 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:11:24.747205 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.700445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:24.747205 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.700565 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:24.747205 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.700614 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret podName:5b982417-f3a9-43ff-9a76-920bd553524e nodeName:}" failed. No retries permitted until 2026-07-08 19:11:25.700600775 +0000 UTC m=+3.002904921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret") pod "global-pull-secret-syncer-6qs6k" (UID: "5b982417-f3a9-43ff-9a76-920bd553524e") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:24.747205 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.729495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:24.863041 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.800689 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:24.863041 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.800784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssw8\" (UniqueName: \"kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8\") pod \"network-check-target-b27qk\" (UID: \"429a71ea-4e0e-4573-8394-bcbe65957d13\") " pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:24.863041 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.800880 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:24.863041 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.800893 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:11:24.863041 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.800912 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:11:24.863041 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.800927 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6ssw8 for pod openshift-network-diagnostics/network-check-target-b27qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:24.863041 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.800952 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs podName:8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:25.80093297 +0000 UTC m=+3.103237133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs") pod "network-metrics-daemon-nfnnj" (UID: "8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:24.863041 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:24.800970 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8 podName:429a71ea-4e0e-4573-8394-bcbe65957d13 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:25.800960954 +0000 UTC m=+3.103265116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ssw8" (UniqueName: "kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8") pod "network-check-target-b27qk" (UID: "429a71ea-4e0e-4573-8394-bcbe65957d13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:24.913232 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.913167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-crrcv" Jul 08 19:11:24.921629 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.921607 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zzbhq" Jul 08 19:11:24.938206 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.938186 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j5pbq" Jul 08 19:11:24.954142 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.954122 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lldql" Jul 08 19:11:24.966708 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.966207 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" Jul 08 19:11:24.976070 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.976049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-26h9r" Jul 08 19:11:24.989198 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:24.989167 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe8542469e4bef4aeca33118acda14c.slice/crio-fc1e063df2244a075c656445d237032f78dfae9167d4f01be8957e50dfc0376c WatchSource:0}: Error finding container fc1e063df2244a075c656445d237032f78dfae9167d4f01be8957e50dfc0376c: Status 404 returned error can't find the container with id fc1e063df2244a075c656445d237032f78dfae9167d4f01be8957e50dfc0376c Jul 08 19:11:24.990165 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:24.990141 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5117d2d045a1c775b90c668c570fae47.slice/crio-9572fb0f5194fc1fe3ceb090f5d160491877411185fdb0915cb832b343797875 WatchSource:0}: Error finding container 9572fb0f5194fc1fe3ceb090f5d160491877411185fdb0915cb832b343797875: Status 404 returned error can't find the container with id 9572fb0f5194fc1fe3ceb090f5d160491877411185fdb0915cb832b343797875 Jul 08 19:11:24.990706 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:24.990678 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7766fe7_0b38_4992_91e4_4079502ca421.slice/crio-d9fbf7873ce8116e44c74e2f3a45b1debacf3066bb746421478623268fd76cab WatchSource:0}: Error finding container d9fbf7873ce8116e44c74e2f3a45b1debacf3066bb746421478623268fd76cab: Status 404 returned error can't find the container with id d9fbf7873ce8116e44c74e2f3a45b1debacf3066bb746421478623268fd76cab Jul 08 19:11:24.991659 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:24.991635 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd385dcd3_5579_42a7_af3d_c9f272c9a568.slice/crio-a52b5381fecea15458ff8b254e2c824ec25a7b575f4f4bd41aec70617942534e WatchSource:0}: Error finding container a52b5381fecea15458ff8b254e2c824ec25a7b575f4f4bd41aec70617942534e: Status 404 returned error can't find the container with id a52b5381fecea15458ff8b254e2c824ec25a7b575f4f4bd41aec70617942534e Jul 08 19:11:24.992337 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:24.992313 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd777b857_7691_4edc_b186_2170dde274d3.slice/crio-ba8646815fc12cfe9b5bf3716756851b0d58c98a0c28e827a71d0fa1dc8c8ac4 WatchSource:0}: Error finding container ba8646815fc12cfe9b5bf3716756851b0d58c98a0c28e827a71d0fa1dc8c8ac4: Status 404 returned error can't find the container with id ba8646815fc12cfe9b5bf3716756851b0d58c98a0c28e827a71d0fa1dc8c8ac4 Jul 08 19:11:24.992550 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.992529 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d48zc" Jul 08 19:11:24.996117 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.996046 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:11:24.999228 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:24.999209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:25.001634 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:25.001611 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94d3f2a_be4c_48b7_b12b_05ac1927e767.slice/crio-0cb931c6f160e3f0a8bd57f0afc3b0707b37037e6a93253587fb7d95e8ac9153 WatchSource:0}: Error finding container 0cb931c6f160e3f0a8bd57f0afc3b0707b37037e6a93253587fb7d95e8ac9153: Status 404 returned error can't find the container with id 0cb931c6f160e3f0a8bd57f0afc3b0707b37037e6a93253587fb7d95e8ac9153 Jul 08 19:11:25.008111 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:25.008082 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a6b211_27a3_4086_914f_cfb2a1418f9b.slice/crio-c83457ee14cb2eb6b6277a70f80bfb6c446daf710c3c51aab93639ed048d84a9 WatchSource:0}: Error finding container c83457ee14cb2eb6b6277a70f80bfb6c446daf710c3c51aab93639ed048d84a9: Status 404 returned error can't find the container with id c83457ee14cb2eb6b6277a70f80bfb6c446daf710c3c51aab93639ed048d84a9 Jul 08 19:11:25.142772 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.142720 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 19:06:24 +0000 UTC" deadline="2028-02-17 05:28:26.27384429 +0000 UTC" Jul 08 19:11:25.142772 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.142767 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14122h17m1.131084299s" Jul 08 19:11:25.269210 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.269101 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lldql" event={"ID":"d385dcd3-5579-42a7-af3d-c9f272c9a568","Type":"ContainerStarted","Data":"a52b5381fecea15458ff8b254e2c824ec25a7b575f4f4bd41aec70617942534e"} Jul 08 19:11:25.269210 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.269208 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:25.270074 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:25.269294 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:25.270074 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.269912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zqct5" event={"ID":"59a6b211-27a3-4086-914f-cfb2a1418f9b","Type":"ContainerStarted","Data":"c83457ee14cb2eb6b6277a70f80bfb6c446daf710c3c51aab93639ed048d84a9"} Jul 08 19:11:25.271121 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.271100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d48zc" event={"ID":"e94d3f2a-be4c-48b7-b12b-05ac1927e767","Type":"ContainerStarted","Data":"0cb931c6f160e3f0a8bd57f0afc3b0707b37037e6a93253587fb7d95e8ac9153"} Jul 08 19:11:25.272339 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.272317 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-26h9r" event={"ID":"d7766fe7-0b38-4992-91e4-4079502ca421","Type":"ContainerStarted","Data":"d9fbf7873ce8116e44c74e2f3a45b1debacf3066bb746421478623268fd76cab"} Jul 08 19:11:25.274580 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.274517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" event={"ID":"d777b857-7691-4edc-b186-2170dde274d3","Type":"ContainerStarted","Data":"ba8646815fc12cfe9b5bf3716756851b0d58c98a0c28e827a71d0fa1dc8c8ac4"} Jul 08 19:11:25.276192 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.276166 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" event={"ID":"cfe8542469e4bef4aeca33118acda14c","Type":"ContainerStarted","Data":"fc1e063df2244a075c656445d237032f78dfae9167d4f01be8957e50dfc0376c"} Jul 08 19:11:25.277170 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.277150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal" event={"ID":"5117d2d045a1c775b90c668c570fae47","Type":"ContainerStarted","Data":"9572fb0f5194fc1fe3ceb090f5d160491877411185fdb0915cb832b343797875"} Jul 08 19:11:25.333213 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:25.333186 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d27b693_5d3b_4cd2_b7e4_9ab67506e2ca.slice/crio-a9f138e622c6ecf8858893719f3905ec41ac8002817a169c3f4d61d9fb4c9b5d WatchSource:0}: Error finding container a9f138e622c6ecf8858893719f3905ec41ac8002817a169c3f4d61d9fb4c9b5d: Status 404 returned error can't find the container with id a9f138e622c6ecf8858893719f3905ec41ac8002817a169c3f4d61d9fb4c9b5d Jul 08 19:11:25.425992 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:25.425964 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1627836b_2975_43af_b809_0476207e62b2.slice/crio-07cbb9d84e8e3d3aaec04b47aa113329e8068171a296a52117adb38c3cf23e19 WatchSource:0}: Error finding container 07cbb9d84e8e3d3aaec04b47aa113329e8068171a296a52117adb38c3cf23e19: Status 404 returned error can't find the container with id 07cbb9d84e8e3d3aaec04b47aa113329e8068171a296a52117adb38c3cf23e19 Jul 08 19:11:25.427902 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:25.427879 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043c26a6_ec7b_4054_9107_e51a93ad9536.slice/crio-f7e3b50fbbd1c6cbe1c8cbe2d3677b6152d60043e64e59967c0d30fe7289518d WatchSource:0}: Error finding container f7e3b50fbbd1c6cbe1c8cbe2d3677b6152d60043e64e59967c0d30fe7289518d: Status 404 returned error can't find the container with id f7e3b50fbbd1c6cbe1c8cbe2d3677b6152d60043e64e59967c0d30fe7289518d Jul 08 19:11:25.434564 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:25.434540 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b08b02_b9a1_4d19_836a_5cc004fb0c0d.slice/crio-7b42cf84af83972fbff10b8abb187bc2b47b5ccdeb9d69113a3dc509bbfe39c6 WatchSource:0}: Error finding container 7b42cf84af83972fbff10b8abb187bc2b47b5ccdeb9d69113a3dc509bbfe39c6: Status 404 returned error can't find the container with id 7b42cf84af83972fbff10b8abb187bc2b47b5ccdeb9d69113a3dc509bbfe39c6 Jul 08 19:11:25.521147 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.521034 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:11:25.707676 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.707624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:25.707880 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:25.707803 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:25.707880 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:25.707870 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret podName:5b982417-f3a9-43ff-9a76-920bd553524e nodeName:}" failed. No retries permitted until 2026-07-08 19:11:27.707856377 +0000 UTC m=+5.010160527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret") pod "global-pull-secret-syncer-6qs6k" (UID: "5b982417-f3a9-43ff-9a76-920bd553524e") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:25.808834 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.808748 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssw8\" (UniqueName: \"kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8\") pod \"network-check-target-b27qk\" (UID: \"429a71ea-4e0e-4573-8394-bcbe65957d13\") " pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:25.809000 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:25.808838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:25.809000 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:25.808855 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:11:25.809000 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:25.808880 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:11:25.809000 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:25.808894 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6ssw8 for pod openshift-network-diagnostics/network-check-target-b27qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:25.809000 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:25.808967 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8 podName:429a71ea-4e0e-4573-8394-bcbe65957d13 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:27.808948074 +0000 UTC m=+5.111252244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ssw8" (UniqueName: "kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8") pod "network-check-target-b27qk" (UID: "429a71ea-4e0e-4573-8394-bcbe65957d13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:25.809000 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:25.808977 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:25.809336 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:25.809019 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs podName:8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:27.809005503 +0000 UTC m=+5.111309664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs") pod "network-metrics-daemon-nfnnj" (UID: "8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:26.143528 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:26.143436 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 19:06:24 +0000 UTC" deadline="2028-02-14 11:37:10.890703122 +0000 UTC" Jul 08 19:11:26.143528 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:26.143480 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14056h25m44.747228405s" Jul 08 19:11:26.262743 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:26.262613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:26.262918 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:26.262775 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:26.263554 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:26.263229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:26.263554 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:26.263337 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:26.300679 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:26.300494 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5pbq" event={"ID":"1627836b-2975-43af-b809-0476207e62b2","Type":"ContainerStarted","Data":"07cbb9d84e8e3d3aaec04b47aa113329e8068171a296a52117adb38c3cf23e19"} Jul 08 19:11:26.312289 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:26.312258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zzbhq" event={"ID":"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca","Type":"ContainerStarted","Data":"a9f138e622c6ecf8858893719f3905ec41ac8002817a169c3f4d61d9fb4c9b5d"} Jul 08 19:11:26.315160 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:26.315131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" event={"ID":"36b08b02-b9a1-4d19-836a-5cc004fb0c0d","Type":"ContainerStarted","Data":"7b42cf84af83972fbff10b8abb187bc2b47b5ccdeb9d69113a3dc509bbfe39c6"} Jul 08 19:11:26.341270 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:26.341235 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crrcv" event={"ID":"043c26a6-ec7b-4054-9107-e51a93ad9536","Type":"ContainerStarted","Data":"f7e3b50fbbd1c6cbe1c8cbe2d3677b6152d60043e64e59967c0d30fe7289518d"} Jul 08 19:11:27.262815 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:27.262780 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:27.262988 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:27.262919 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:27.727466 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:27.726792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:27.727466 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:27.726964 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:27.727466 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:27.727024 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret podName:5b982417-f3a9-43ff-9a76-920bd553524e nodeName:}" failed. No retries permitted until 2026-07-08 19:11:31.72700631 +0000 UTC m=+9.029310460 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret") pod "global-pull-secret-syncer-6qs6k" (UID: "5b982417-f3a9-43ff-9a76-920bd553524e") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:27.827940 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:27.827902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssw8\" (UniqueName: \"kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8\") pod \"network-check-target-b27qk\" (UID: \"429a71ea-4e0e-4573-8394-bcbe65957d13\") " pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:27.828135 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:27.827971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:27.828135 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:27.828115 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:27.828252 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:27.828178 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs podName:8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:31.828155948 +0000 UTC m=+9.130460094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs") pod "network-metrics-daemon-nfnnj" (UID: "8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:27.828649 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:27.828603 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:11:27.828649 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:27.828626 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:11:27.828649 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:27.828642 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6ssw8 for pod openshift-network-diagnostics/network-check-target-b27qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:27.828902 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:27.828691 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8 podName:429a71ea-4e0e-4573-8394-bcbe65957d13 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:31.828674849 +0000 UTC m=+9.130979010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ssw8" (UniqueName: "kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8") pod "network-check-target-b27qk" (UID: "429a71ea-4e0e-4573-8394-bcbe65957d13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:28.262401 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:28.262368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:28.262585 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:28.262495 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:28.262585 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:28.262580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:28.262732 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:28.262672 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:29.262653 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:29.262615 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:29.263203 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:29.262758 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:30.261712 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:30.261621 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:30.261917 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:30.261773 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:30.262255 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:30.262226 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:30.262393 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:30.262371 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:31.262395 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:31.262067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:31.262395 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:31.262220 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:31.763989 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:31.763943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:31.764183 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:31.764166 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:31.764246 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:31.764234 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret podName:5b982417-f3a9-43ff-9a76-920bd553524e nodeName:}" failed. No retries permitted until 2026-07-08 19:11:39.764217559 +0000 UTC m=+17.066521705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret") pod "global-pull-secret-syncer-6qs6k" (UID: "5b982417-f3a9-43ff-9a76-920bd553524e") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:31.865877 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:31.865025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:31.865877 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:31.865102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssw8\" (UniqueName: \"kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8\") pod \"network-check-target-b27qk\" (UID: \"429a71ea-4e0e-4573-8394-bcbe65957d13\") " pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:31.865877 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:31.865266 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:11:31.865877 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:31.865287 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:11:31.865877 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:31.865299 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6ssw8 for pod openshift-network-diagnostics/network-check-target-b27qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:31.865877 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:31.865368 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8 podName:429a71ea-4e0e-4573-8394-bcbe65957d13 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:39.865336346 +0000 UTC m=+17.167640509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ssw8" (UniqueName: "kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8") pod "network-check-target-b27qk" (UID: "429a71ea-4e0e-4573-8394-bcbe65957d13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:31.865877 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:31.865796 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:31.865877 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:31.865844 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs podName:8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:39.865829646 +0000 UTC m=+17.168133793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs") pod "network-metrics-daemon-nfnnj" (UID: "8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:32.262724 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:32.262689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:32.263196 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:32.262812 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:32.263196 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:32.263161 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:32.263315 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:32.263261 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:33.263357 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:33.263309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:33.263790 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:33.263436 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:34.262098 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:34.262064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:34.262289 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:34.262210 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:34.262520 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:34.262496 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:34.262666 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:34.262639 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:35.263151 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:35.262177 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:35.263151 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:35.262321 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:36.262027 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:36.261986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:36.262027 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:36.261986 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:36.262254 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:36.262115 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:36.262301 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:36.262247 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:37.262476 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:37.262445 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:37.262919 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:37.262574 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:38.262386 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:38.262356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:38.262580 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:38.262396 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:38.262580 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:38.262492 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:38.262967 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:38.262616 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:39.262041 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:39.261999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:39.262235 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:39.262134 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:39.827672 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:39.826527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:39.827672 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:39.826731 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:39.827672 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:39.826806 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret podName:5b982417-f3a9-43ff-9a76-920bd553524e nodeName:}" failed. No retries permitted until 2026-07-08 19:11:55.826785837 +0000 UTC m=+33.129089984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret") pod "global-pull-secret-syncer-6qs6k" (UID: "5b982417-f3a9-43ff-9a76-920bd553524e") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:39.927224 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:39.927187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssw8\" (UniqueName: \"kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8\") pod \"network-check-target-b27qk\" (UID: \"429a71ea-4e0e-4573-8394-bcbe65957d13\") " pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:39.927415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:39.927260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:39.927415 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:39.927375 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:39.927415 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:39.927397 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:11:39.927554 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:39.927425 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:11:39.927554 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:39.927440 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6ssw8 for pod openshift-network-diagnostics/network-check-target-b27qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:39.927554 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:39.927446 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs podName:8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:55.927424308 +0000 UTC m=+33.229728453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs") pod "network-metrics-daemon-nfnnj" (UID: "8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:39.927554 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:39.927491 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8 podName:429a71ea-4e0e-4573-8394-bcbe65957d13 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:55.927475646 +0000 UTC m=+33.229779797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ssw8" (UniqueName: "kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8") pod "network-check-target-b27qk" (UID: "429a71ea-4e0e-4573-8394-bcbe65957d13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:40.261905 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:40.261822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:40.262100 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:40.261844 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:40.262100 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:40.261953 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:40.262100 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:40.262000 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:41.262460 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:41.262421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:41.262958 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:41.262554 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:42.261827 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:42.261792 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:42.262028 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:42.261792 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:42.262028 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:42.261939 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:42.262148 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:42.262022 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:43.262190 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:43.262152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:43.262641 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:43.262247 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:44.262282 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:44.262250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:44.262742 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:44.262387 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:44.262742 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:44.262452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:44.262742 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:44.262568 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:45.262016 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:45.261977 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:45.262187 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:45.262118 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:46.261939 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.261656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:46.262421 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.261687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:46.262421 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:46.262037 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:46.262421 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:46.262090 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:46.395952 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.395918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j5pbq" event={"ID":"1627836b-2975-43af-b809-0476207e62b2","Type":"ContainerStarted","Data":"09131fb56287858d630cb4136abd322f98f9b7da290fab5e46d601aee6ac75f8"} Jul 08 19:11:46.399205 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.399180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d48zc" event={"ID":"e94d3f2a-be4c-48b7-b12b-05ac1927e767","Type":"ContainerStarted","Data":"9b2eebc46e779ed157cb79e081a627afcf4faabef99abe7c89a3c8bd45168ca3"} Jul 08 19:11:46.401210 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.401189 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal" event={"ID":"5117d2d045a1c775b90c668c570fae47","Type":"ContainerStarted","Data":"89efb55ff54ad3dbb2b2a8f3a36dab8b80de0bba31f4d1cabb3581ad62eb1a1a"} Jul 08 19:11:46.404197 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.404179 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:11:46.404579 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.404547 2576 generic.go:356] "Generic (PLEG): container finished" podID="36b08b02-b9a1-4d19-836a-5cc004fb0c0d" containerID="662c9e64cac0acbe8753c197c9842c94f72712f98bca55d3f331387f80c3774a" exitCode=1 Jul 08 19:11:46.404684 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.404585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" event={"ID":"36b08b02-b9a1-4d19-836a-5cc004fb0c0d","Type":"ContainerStarted","Data":"334f8cfc60b5a6eccb10b4e803af843218562d006479ce774a8c537893f5567f"} Jul 08 19:11:46.404684 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.404632 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" event={"ID":"36b08b02-b9a1-4d19-836a-5cc004fb0c0d","Type":"ContainerStarted","Data":"4f1eaa393a51b7894805cf86a719a8b4bc0d9c3881a98d685f25137fe87eb61c"} Jul 08 19:11:46.404684 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.404661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" event={"ID":"36b08b02-b9a1-4d19-836a-5cc004fb0c0d","Type":"ContainerStarted","Data":"533f5e534d6b68b374a0e3e0ad9691fdab5ca3f243cf23a4d07ba8370ab345af"} Jul 08 19:11:46.404841 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.404689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" event={"ID":"36b08b02-b9a1-4d19-836a-5cc004fb0c0d","Type":"ContainerStarted","Data":"2213f74599b319f29f608a1e0e1e24301d56e51781b147fe8b7417015d317397"} Jul 08 19:11:46.404841 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.404719 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" event={"ID":"36b08b02-b9a1-4d19-836a-5cc004fb0c0d","Type":"ContainerDied","Data":"662c9e64cac0acbe8753c197c9842c94f72712f98bca55d3f331387f80c3774a"} Jul 08 19:11:46.404841 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.404746 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" event={"ID":"36b08b02-b9a1-4d19-836a-5cc004fb0c0d","Type":"ContainerStarted","Data":"8ba5a6c1d7fd07904d8fae163ce4e526cf6edc7a5bf634411bc970f0689389fb"} Jul 08 19:11:46.418650 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.418609 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j5pbq" podStartSLOduration=2.011938358 podStartE2EDuration="22.418597243s" totalImagesPullingTime="20.406658885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:46.417946536 +0000 UTC m=+23.720250694" watchObservedRunningTime="2026-07-08 19:11:46.418597243 +0000 UTC m=+23.720901409" Jul 08 19:11:46.436310 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.436267 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-188.ec2.internal" podStartSLOduration=22.436255562 podStartE2EDuration="22.436255562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:46.435286306 +0000 UTC m=+23.737590477" watchObservedRunningTime="2026-07-08 19:11:46.436255562 +0000 UTC m=+23.738559729" Jul 08 19:11:46.457865 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:46.457820 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d48zc" podStartSLOduration=1.677442847 podStartE2EDuration="22.457805168s" totalImagesPullingTime="20.780362321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:46.457660681 +0000 UTC m=+23.759964851" watchObservedRunningTime="2026-07-08 19:11:46.457805168 +0000 UTC m=+23.760109336" Jul 08 19:11:47.262890 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.262580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:47.263659 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:47.262913 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:47.408183 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.408152 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" event={"ID":"d777b857-7691-4edc-b186-2170dde274d3","Type":"ContainerStarted","Data":"a3f14fd22a8c1108df5c2ce7fd3c170de06f77f10fadee0dd3c302801e2e313b"} Jul 08 19:11:47.409614 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.409588 2576 generic.go:356] "Generic (PLEG): container finished" podID="cfe8542469e4bef4aeca33118acda14c" containerID="1ac9095196c295b7966652cdd57be0f3d3b65982c2f7ac27371044f46ba8d5c2" exitCode=0 Jul 08 19:11:47.409735 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.409662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" event={"ID":"cfe8542469e4bef4aeca33118acda14c","Type":"ContainerDied","Data":"1ac9095196c295b7966652cdd57be0f3d3b65982c2f7ac27371044f46ba8d5c2"} Jul 08 19:11:47.411079 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.411053 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zzbhq" event={"ID":"1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca","Type":"ContainerStarted","Data":"2162682eafa0ea5635cd38f664c698cdff1540f36d4a3640e2e2c51ea7083b8c"} Jul 08 19:11:47.414359 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.414324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lldql" event={"ID":"d385dcd3-5579-42a7-af3d-c9f272c9a568","Type":"ContainerStarted","Data":"da0db6ea6dc553732888779ba3a9cebb73b04543f62d0f01f91c952cc122a5b7"} Jul 08 19:11:47.415887 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.415868 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-crrcv" event={"ID":"043c26a6-ec7b-4054-9107-e51a93ad9536","Type":"ContainerStarted","Data":"1bb3b845b490184011258737bb6d877b08b5558ca42dace9f94bb8a508ba3e79"} Jul 08 19:11:47.417207 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.417185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zqct5" event={"ID":"59a6b211-27a3-4086-914f-cfb2a1418f9b","Type":"ContainerStarted","Data":"eb3b51dee9356f6ab92cd148cbf9412ce50b14be30ecdca5926c06d5fcf40b38"} Jul 08 19:11:47.418488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.418467 2576 generic.go:356] "Generic (PLEG): container finished" podID="d7766fe7-0b38-4992-91e4-4079502ca421" containerID="8dd0ddcefcc50c23b1e702c44dd43cebb37f7878ef0f128e22e5ce12c1e9335a" exitCode=0 Jul 08 19:11:47.418562 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.418532 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-26h9r" event={"ID":"d7766fe7-0b38-4992-91e4-4079502ca421","Type":"ContainerDied","Data":"8dd0ddcefcc50c23b1e702c44dd43cebb37f7878ef0f128e22e5ce12c1e9335a"} Jul 08 19:11:47.475786 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.475740 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-crrcv" podStartSLOduration=4.126120741 podStartE2EDuration="24.47572453s" totalImagesPullingTime="20.349603789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:47.475578426 +0000 UTC m=+24.777882594" watchObservedRunningTime="2026-07-08 19:11:47.47572453 +0000 UTC m=+24.778028698" Jul 08 19:11:47.516584 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.516513 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zzbhq" podStartSLOduration=4.067026018 podStartE2EDuration="24.516498545s" totalImagesPullingTime="20.449472527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:47.500045717 +0000 UTC m=+24.802349884" watchObservedRunningTime="2026-07-08 19:11:47.516498545 +0000 UTC m=+24.818802795" Jul 08 19:11:47.516584 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.516578 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lldql" podStartSLOduration=2.733425656 podStartE2EDuration="23.516573828s" totalImagesPullingTime="20.783148172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:47.515995255 +0000 UTC m=+24.818299424" watchObservedRunningTime="2026-07-08 19:11:47.516573828 +0000 UTC m=+24.818878067" Jul 08 19:11:47.567219 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.567162 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zqct5" podStartSLOduration=2.797434687 podStartE2EDuration="23.567141983s" totalImagesPullingTime="20.769707296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:47.566420865 +0000 UTC m=+24.868725033" watchObservedRunningTime="2026-07-08 19:11:47.567141983 +0000 UTC m=+24.869446153" Jul 08 19:11:47.996213 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:47.996188 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 19:11:48.155385 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:48.155229 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T19:11:47.99620956Z","UUID":"41a66cd4-ab36-4ddf-9971-7ffaacab8efd","Handler":null,"Name":"","Endpoint":""} Jul 08 19:11:48.156635 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:48.156613 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 19:11:48.156748 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:48.156649 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 19:11:48.262412 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:48.262380 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:48.262585 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:48.262420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:48.262585 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:48.262508 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:48.262697 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:48.262621 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:48.422386 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:48.422337 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" event={"ID":"d777b857-7691-4edc-b186-2170dde274d3","Type":"ContainerStarted","Data":"ba753cc2255aaf1dcf53b11c2c666c97f8dc883acf848c04c9e9684af69edd67"} Jul 08 19:11:48.424397 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:48.424368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" event={"ID":"cfe8542469e4bef4aeca33118acda14c","Type":"ContainerStarted","Data":"b276d2e6c36457f7e9af79741882c4956339f9c8250433cf38dc4138e6d155ec"} Jul 08 19:11:48.470159 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:48.470118 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-188.ec2.internal" podStartSLOduration=24.470101991 podStartE2EDuration="24.470101991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:48.469460176 +0000 UTC m=+25.771764345" watchObservedRunningTime="2026-07-08 19:11:48.470101991 +0000 UTC m=+25.772406159" Jul 08 19:11:49.262640 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:49.262604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:49.262822 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:49.262738 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:49.430185 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:49.430160 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:11:49.430704 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:49.430679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" event={"ID":"36b08b02-b9a1-4d19-836a-5cc004fb0c0d","Type":"ContainerStarted","Data":"698f47b1494a03df937b207b93ab2ca1e945ee3b1a2bdd5c3f722d2b851b51d5"} Jul 08 19:11:50.261839 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:50.261771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:50.261978 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:50.261874 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:50.261978 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:50.261905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:50.261978 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:50.261965 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:50.435131 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:50.435097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" event={"ID":"d777b857-7691-4edc-b186-2170dde274d3","Type":"ContainerStarted","Data":"a982d033f0861d3f41b3504795fcf7a9ffcc66e3d818fc3829a502c5eb835180"} Jul 08 19:11:50.454353 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:50.454303 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b94vd" podStartSLOduration=6.234960421 podStartE2EDuration="26.454287561s" totalImagesPullingTime="20.21932714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:50.453911088 +0000 UTC m=+27.756215258" watchObservedRunningTime="2026-07-08 19:11:50.454287561 +0000 UTC m=+27.756591774" Jul 08 19:11:51.262367 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:51.262189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:51.262577 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:51.262467 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:51.439893 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:51.439866 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:11:51.440415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:51.440257 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" event={"ID":"36b08b02-b9a1-4d19-836a-5cc004fb0c0d","Type":"ContainerStarted","Data":"c439cb177ec2b852f3822ff6a9a1b30f495b02dd706f91af591e50a6d12e0dca"} Jul 08 19:11:51.440799 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:51.440779 2576 scope.go:117] "RemoveContainer" containerID="662c9e64cac0acbe8753c197c9842c94f72712f98bca55d3f331387f80c3774a" Jul 08 19:11:51.658639 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:51.658613 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:51.659124 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:51.659108 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:52.262087 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.262063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:52.262226 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.262063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:52.262226 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:52.262166 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:52.262303 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:52.262237 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:52.446126 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.446099 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:11:52.446858 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.446464 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" event={"ID":"36b08b02-b9a1-4d19-836a-5cc004fb0c0d","Type":"ContainerStarted","Data":"b8a43be247beb7fe182f824405512ba4856e6bcf040a993f99e182342de06ca6"} Jul 08 19:11:52.446858 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.446807 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:52.447011 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.446995 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:52.447077 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.447065 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:52.447133 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.447120 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:52.447234 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.447210 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zqct5" Jul 08 19:11:52.462419 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.462398 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:52.462754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.462739 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:11:52.482601 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.482556 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" podStartSLOduration=9.076834422 podStartE2EDuration="29.482539504s" totalImagesPullingTime="20.405705082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:11:52.480729048 +0000 UTC m=+29.783033227" watchObservedRunningTime="2026-07-08 19:11:52.482539504 +0000 UTC m=+29.784843670" Jul 08 19:11:52.846640 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.846599 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nfnnj"] Jul 08 19:11:52.846827 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.846725 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:52.846888 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:52.846840 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:52.850658 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.850634 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6qs6k"] Jul 08 19:11:52.850827 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.850726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:52.850883 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:52.850821 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:52.851296 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.851271 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b27qk"] Jul 08 19:11:52.851404 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:52.851389 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:52.851504 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:52.851478 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:54.262609 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:54.262417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:54.263027 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:54.262441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:54.263027 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:54.262688 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:54.263027 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:54.262805 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:55.262498 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:55.262463 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:55.262663 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:55.262606 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:55.849267 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:55.849188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:55.849429 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:55.849334 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:55.849429 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:55.849406 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret podName:5b982417-f3a9-43ff-9a76-920bd553524e nodeName:}" failed. No retries permitted until 2026-07-08 19:12:27.849391559 +0000 UTC m=+65.151695703 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret") pod "global-pull-secret-syncer-6qs6k" (UID: "5b982417-f3a9-43ff-9a76-920bd553524e") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:11:55.950336 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:55.950309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:55.950480 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:55.950367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssw8\" (UniqueName: \"kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8\") pod \"network-check-target-b27qk\" (UID: \"429a71ea-4e0e-4573-8394-bcbe65957d13\") " pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:55.950534 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:55.950477 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:55.950534 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:55.950523 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:11:55.950640 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:55.950538 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:11:55.950640 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:55.950543 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs podName:8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:27.950522989 +0000 UTC m=+65.252827149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs") pod "network-metrics-daemon-nfnnj" (UID: "8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:11:55.950640 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:55.950547 2576 projected.go:196] Error preparing data for projected volume kube-api-access-6ssw8 for pod openshift-network-diagnostics/network-check-target-b27qk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:55.950640 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:55.950587 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8 podName:429a71ea-4e0e-4573-8394-bcbe65957d13 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:27.950576438 +0000 UTC m=+65.252880610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6ssw8" (UniqueName: "kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8") pod "network-check-target-b27qk" (UID: "429a71ea-4e0e-4573-8394-bcbe65957d13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:11:56.261903 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:56.261830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:56.262030 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:56.261927 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:56.262030 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:56.261978 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:56.262109 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:56.262062 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:56.457382 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:56.457331 2576 generic.go:356] "Generic (PLEG): container finished" podID="d7766fe7-0b38-4992-91e4-4079502ca421" containerID="26a1275ee36f66bbc90449f4cc63f11cbc3e7b9f00d4b681b4bfe1b506c8de00" exitCode=0 Jul 08 19:11:56.457888 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:56.457416 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-26h9r" event={"ID":"d7766fe7-0b38-4992-91e4-4079502ca421","Type":"ContainerDied","Data":"26a1275ee36f66bbc90449f4cc63f11cbc3e7b9f00d4b681b4bfe1b506c8de00"} Jul 08 19:11:57.261814 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:57.261721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:57.261958 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:57.261898 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nfnnj" podUID="8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6" Jul 08 19:11:57.461708 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:57.461680 2576 generic.go:356] "Generic (PLEG): container finished" podID="d7766fe7-0b38-4992-91e4-4079502ca421" containerID="c0c902cc56c4fdc05b4fdf90ca96468ce47fed20916aade50ed9b6e99745dfce" exitCode=0 Jul 08 19:11:57.462014 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:57.461742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-26h9r" event={"ID":"d7766fe7-0b38-4992-91e4-4079502ca421","Type":"ContainerDied","Data":"c0c902cc56c4fdc05b4fdf90ca96468ce47fed20916aade50ed9b6e99745dfce"} Jul 08 19:11:58.261906 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.261606 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:11:58.261906 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:58.261730 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b27qk" podUID="429a71ea-4e0e-4573-8394-bcbe65957d13" Jul 08 19:11:58.261906 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.261786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:11:58.261906 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:58.261892 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qs6k" podUID="5b982417-f3a9-43ff-9a76-920bd553524e" Jul 08 19:11:58.466228 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.466005 2576 generic.go:356] "Generic (PLEG): container finished" podID="d7766fe7-0b38-4992-91e4-4079502ca421" containerID="632abff1b2ea5f7a0bbbdebcefd8ac331cd3d827069e3fe445add1aa2ff81fc9" exitCode=0 Jul 08 19:11:58.466228 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.466078 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-26h9r" event={"ID":"d7766fe7-0b38-4992-91e4-4079502ca421","Type":"ContainerDied","Data":"632abff1b2ea5f7a0bbbdebcefd8ac331cd3d827069e3fe445add1aa2ff81fc9"} Jul 08 19:11:58.469100 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.469080 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-188.ec2.internal" event="NodeReady" Jul 08 19:11:58.469243 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.469232 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 19:11:58.501595 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.501562 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5ff55887b9-vn4sp"] Jul 08 19:11:58.503464 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.503448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.506029 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.506007 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 19:11:58.506154 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.506074 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 19:11:58.506217 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.506201 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 19:11:58.506483 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.506470 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9x74r\"" Jul 08 19:11:58.511748 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.511728 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 19:11:58.516184 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.516167 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8njxt"] Jul 08 19:11:58.517865 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.517853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.520288 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.520267 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ff55887b9-vn4sp"] Jul 08 19:11:58.520661 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.520644 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 19:11:58.520761 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.520728 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4xh72\"" Jul 08 19:11:58.520833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.520659 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 19:11:58.520935 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.520918 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pbwkx"] Jul 08 19:11:58.523414 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.523395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pbwkx" Jul 08 19:11:58.526235 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.526221 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bfkrp\"" Jul 08 19:11:58.526316 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.526220 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 19:11:58.526434 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.526418 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 19:11:58.526493 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.526481 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 19:11:58.541753 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.541731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8njxt"] Jul 08 19:11:58.542513 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.542485 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pbwkx"] Jul 08 19:11:58.666136 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.666136 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-installation-pull-secrets\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.666415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666154 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz8vl\" (UniqueName: \"kubernetes.io/projected/3db481d0-9cdf-46be-a738-505e054ba95b-kube-api-access-nz8vl\") pod \"ingress-canary-pbwkx\" (UID: \"3db481d0-9cdf-46be-a738-505e054ba95b\") " pod="openshift-ingress-canary/ingress-canary-pbwkx" Jul 08 19:11:58.666415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.666415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r295j\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-kube-api-access-r295j\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.666415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53d10eff-78be-422c-b7ce-13d970eca76d-config-volume\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.666415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9z2\" (UniqueName: \"kubernetes.io/projected/53d10eff-78be-422c-b7ce-13d970eca76d-kube-api-access-gw9z2\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.666670 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666439 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-ca-trust-extracted\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.666670 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-certificates\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.666670 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-bound-sa-token\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.666670 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666515 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3db481d0-9cdf-46be-a738-505e054ba95b-cert\") pod \"ingress-canary-pbwkx\" (UID: \"3db481d0-9cdf-46be-a738-505e054ba95b\") " pod="openshift-ingress-canary/ingress-canary-pbwkx" Jul 08 19:11:58.666670 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-image-registry-private-configuration\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.666670 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-trusted-ca\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.666670 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.666615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d10eff-78be-422c-b7ce-13d970eca76d-tmp-dir\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.767526 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.767526 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-installation-pull-secrets\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.767526 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz8vl\" (UniqueName: \"kubernetes.io/projected/3db481d0-9cdf-46be-a738-505e054ba95b-kube-api-access-nz8vl\") pod \"ingress-canary-pbwkx\" (UID: \"3db481d0-9cdf-46be-a738-505e054ba95b\") " pod="openshift-ingress-canary/ingress-canary-pbwkx" Jul 08 19:11:58.767785 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.767785 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r295j\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-kube-api-access-r295j\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.767785 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:58.767588 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:11:58.767785 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:58.767620 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff55887b9-vn4sp: secret "image-registry-tls" not found Jul 08 19:11:58.767785 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:58.767689 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls podName:1cad1733-48a4-4a9b-bd22-fb2e7b35c008 nodeName:}" failed. No retries permitted until 2026-07-08 19:11:59.267673006 +0000 UTC m=+36.569977164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls") pod "image-registry-5ff55887b9-vn4sp" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008") : secret "image-registry-tls" not found Jul 08 19:11:58.767785 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53d10eff-78be-422c-b7ce-13d970eca76d-config-volume\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.767785 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9z2\" (UniqueName: \"kubernetes.io/projected/53d10eff-78be-422c-b7ce-13d970eca76d-kube-api-access-gw9z2\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.767785 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:58.767772 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:11:58.768172 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-ca-trust-extracted\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.768172 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:58.767831 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls podName:53d10eff-78be-422c-b7ce-13d970eca76d nodeName:}" failed. No retries permitted until 2026-07-08 19:11:59.26781412 +0000 UTC m=+36.570118270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls") pod "dns-default-8njxt" (UID: "53d10eff-78be-422c-b7ce-13d970eca76d") : secret "dns-default-metrics-tls" not found Jul 08 19:11:58.768172 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-certificates\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.768172 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-bound-sa-token\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.768172 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3db481d0-9cdf-46be-a738-505e054ba95b-cert\") pod \"ingress-canary-pbwkx\" (UID: \"3db481d0-9cdf-46be-a738-505e054ba95b\") " pod="openshift-ingress-canary/ingress-canary-pbwkx" Jul 08 19:11:58.768172 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-image-registry-private-configuration\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.768172 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.767999 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-trusted-ca\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.768172 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.768022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d10eff-78be-422c-b7ce-13d970eca76d-tmp-dir\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.768566 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.768190 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/53d10eff-78be-422c-b7ce-13d970eca76d-config-volume\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.768566 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.768242 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-ca-trust-extracted\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.768566 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.768311 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53d10eff-78be-422c-b7ce-13d970eca76d-tmp-dir\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.768795 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.768776 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-certificates\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.769062 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.769043 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-trusted-ca\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.771900 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.771881 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-installation-pull-secrets\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.771965 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.771881 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-image-registry-private-configuration\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.771965 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.771948 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3db481d0-9cdf-46be-a738-505e054ba95b-cert\") pod \"ingress-canary-pbwkx\" (UID: \"3db481d0-9cdf-46be-a738-505e054ba95b\") " pod="openshift-ingress-canary/ingress-canary-pbwkx" Jul 08 19:11:58.776689 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.776653 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r295j\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-kube-api-access-r295j\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.777018 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.776996 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz8vl\" (UniqueName: \"kubernetes.io/projected/3db481d0-9cdf-46be-a738-505e054ba95b-kube-api-access-nz8vl\") pod \"ingress-canary-pbwkx\" (UID: \"3db481d0-9cdf-46be-a738-505e054ba95b\") " pod="openshift-ingress-canary/ingress-canary-pbwkx" Jul 08 19:11:58.777098 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.777055 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9z2\" (UniqueName: \"kubernetes.io/projected/53d10eff-78be-422c-b7ce-13d970eca76d-kube-api-access-gw9z2\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:58.777423 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.777401 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-bound-sa-token\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:58.832921 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.832884 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pbwkx" Jul 08 19:11:58.982121 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:58.982088 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pbwkx"] Jul 08 19:11:58.986976 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:11:58.986942 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db481d0_9cdf_46be_a738_505e054ba95b.slice/crio-fdaed6feedc138216975f806bf56a68f938c4021e2282f13868cc3fc918b900e WatchSource:0}: Error finding container fdaed6feedc138216975f806bf56a68f938c4021e2282f13868cc3fc918b900e: Status 404 returned error can't find the container with id fdaed6feedc138216975f806bf56a68f938c4021e2282f13868cc3fc918b900e Jul 08 19:11:59.262249 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:59.262220 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:11:59.265419 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:59.265391 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 19:11:59.265690 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:59.265653 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8qd8q\"" Jul 08 19:11:59.271201 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:59.271177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:11:59.271370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:59.271216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:11:59.271370 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:59.271332 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:11:59.271480 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:59.271405 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls podName:53d10eff-78be-422c-b7ce-13d970eca76d nodeName:}" failed. No retries permitted until 2026-07-08 19:12:00.271388943 +0000 UTC m=+37.573693092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls") pod "dns-default-8njxt" (UID: "53d10eff-78be-422c-b7ce-13d970eca76d") : secret "dns-default-metrics-tls" not found Jul 08 19:11:59.271480 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:59.271438 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:11:59.271480 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:59.271458 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff55887b9-vn4sp: secret "image-registry-tls" not found Jul 08 19:11:59.271626 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:11:59.271520 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls podName:1cad1733-48a4-4a9b-bd22-fb2e7b35c008 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:00.271499975 +0000 UTC m=+37.573804137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls") pod "image-registry-5ff55887b9-vn4sp" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008") : secret "image-registry-tls" not found Jul 08 19:11:59.472360 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:11:59.472310 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pbwkx" event={"ID":"3db481d0-9cdf-46be-a738-505e054ba95b","Type":"ContainerStarted","Data":"fdaed6feedc138216975f806bf56a68f938c4021e2282f13868cc3fc918b900e"} Jul 08 19:12:00.261850 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:00.261812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:12:00.262022 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:00.261812 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:12:00.265079 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:00.265059 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 19:12:00.265197 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:00.265115 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 19:12:00.266296 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:00.266279 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-92l7d\"" Jul 08 19:12:00.266417 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:00.266279 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 19:12:00.279395 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:00.279375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:12:00.279468 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:00.279407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:12:00.279521 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:00.279510 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:12:00.279556 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:00.279518 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:12:00.279556 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:00.279534 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff55887b9-vn4sp: secret "image-registry-tls" not found Jul 08 19:12:00.279617 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:00.279560 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls podName:53d10eff-78be-422c-b7ce-13d970eca76d nodeName:}" failed. No retries permitted until 2026-07-08 19:12:02.279547014 +0000 UTC m=+39.581851158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls") pod "dns-default-8njxt" (UID: "53d10eff-78be-422c-b7ce-13d970eca76d") : secret "dns-default-metrics-tls" not found Jul 08 19:12:00.279617 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:00.279577 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls podName:1cad1733-48a4-4a9b-bd22-fb2e7b35c008 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:02.279565631 +0000 UTC m=+39.581869792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls") pod "image-registry-5ff55887b9-vn4sp" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008") : secret "image-registry-tls" not found Jul 08 19:12:02.295442 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:02.295239 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:12:02.295824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:02.295458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:12:02.295824 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:02.295419 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:12:02.295824 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:02.295544 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff55887b9-vn4sp: secret "image-registry-tls" not found Jul 08 19:12:02.295824 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:02.295621 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls podName:1cad1733-48a4-4a9b-bd22-fb2e7b35c008 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:06.29559544 +0000 UTC m=+43.597899588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls") pod "image-registry-5ff55887b9-vn4sp" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008") : secret "image-registry-tls" not found Jul 08 19:12:02.295824 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:02.295627 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:12:02.295824 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:02.295674 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls podName:53d10eff-78be-422c-b7ce-13d970eca76d nodeName:}" failed. No retries permitted until 2026-07-08 19:12:06.295658776 +0000 UTC m=+43.597962922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls") pod "dns-default-8njxt" (UID: "53d10eff-78be-422c-b7ce-13d970eca76d") : secret "dns-default-metrics-tls" not found Jul 08 19:12:03.484457 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:03.484417 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pbwkx" event={"ID":"3db481d0-9cdf-46be-a738-505e054ba95b","Type":"ContainerStarted","Data":"f0cffb8bc4c50e10f84150d2142adf9391100a0228b03d11651ceb6b7e9d71d3"} Jul 08 19:12:03.506073 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:03.505265 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pbwkx" podStartSLOduration=1.859092799 podStartE2EDuration="5.505245901s" totalImagesPullingTime="3.646153102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:03.502070525 +0000 UTC m=+40.804374693" watchObservedRunningTime="2026-07-08 19:12:03.505245901 +0000 UTC m=+40.807550068" Jul 08 19:12:06.328856 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:06.328813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:12:06.328856 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:06.328861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:12:06.329424 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:06.328996 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:12:06.329424 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:06.329032 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff55887b9-vn4sp: secret "image-registry-tls" not found Jul 08 19:12:06.329424 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:06.329094 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls podName:1cad1733-48a4-4a9b-bd22-fb2e7b35c008 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:14.329075679 +0000 UTC m=+51.631379824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls") pod "image-registry-5ff55887b9-vn4sp" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008") : secret "image-registry-tls" not found Jul 08 19:12:06.329424 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:06.329001 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:12:06.329424 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:06.329148 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls podName:53d10eff-78be-422c-b7ce-13d970eca76d nodeName:}" failed. No retries permitted until 2026-07-08 19:12:14.329136607 +0000 UTC m=+51.631440752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls") pod "dns-default-8njxt" (UID: "53d10eff-78be-422c-b7ce-13d970eca76d") : secret "dns-default-metrics-tls" not found Jul 08 19:12:06.493772 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:06.493732 2576 generic.go:356] "Generic (PLEG): container finished" podID="d7766fe7-0b38-4992-91e4-4079502ca421" containerID="0743c12e21564de1181e4d4fa00126fa7a2b0f53acdd8fdf224cf3cf05973861" exitCode=0 Jul 08 19:12:06.493924 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:06.493793 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-26h9r" event={"ID":"d7766fe7-0b38-4992-91e4-4079502ca421","Type":"ContainerDied","Data":"0743c12e21564de1181e4d4fa00126fa7a2b0f53acdd8fdf224cf3cf05973861"} Jul 08 19:12:07.498813 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:07.498779 2576 generic.go:356] "Generic (PLEG): container finished" podID="d7766fe7-0b38-4992-91e4-4079502ca421" containerID="8a812b6a3b2c552fcd6fcdaad049f08d0220a7eafbc99d5c326d30b12a523d2d" exitCode=0 Jul 08 19:12:07.499188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:07.498837 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-26h9r" event={"ID":"d7766fe7-0b38-4992-91e4-4079502ca421","Type":"ContainerDied","Data":"8a812b6a3b2c552fcd6fcdaad049f08d0220a7eafbc99d5c326d30b12a523d2d"} Jul 08 19:12:08.504015 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:08.503975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-26h9r" event={"ID":"d7766fe7-0b38-4992-91e4-4079502ca421","Type":"ContainerStarted","Data":"fcee1a5602a9ad893a6d47d6aee73402c3d5411d095c76f5f1972667985af4d4"} Jul 08 19:12:08.527844 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:08.527791 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-26h9r" podStartSLOduration=7.11979619 podStartE2EDuration="44.527772704s" totalImagesPullingTime="37.407976514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:08.526044783 +0000 UTC m=+45.828348949" watchObservedRunningTime="2026-07-08 19:12:08.527772704 +0000 UTC m=+45.830076871" Jul 08 19:12:14.391736 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:14.391700 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:12:14.391736 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:14.391733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:12:14.392291 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:14.391834 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:12:14.392291 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:14.391885 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls podName:53d10eff-78be-422c-b7ce-13d970eca76d nodeName:}" failed. No retries permitted until 2026-07-08 19:12:30.39187223 +0000 UTC m=+67.694176374 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls") pod "dns-default-8njxt" (UID: "53d10eff-78be-422c-b7ce-13d970eca76d") : secret "dns-default-metrics-tls" not found Jul 08 19:12:14.392291 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:14.391880 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:12:14.392291 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:14.391905 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff55887b9-vn4sp: secret "image-registry-tls" not found Jul 08 19:12:14.392291 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:14.391984 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls podName:1cad1733-48a4-4a9b-bd22-fb2e7b35c008 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:30.39196459 +0000 UTC m=+67.694268753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls") pod "image-registry-5ff55887b9-vn4sp" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008") : secret "image-registry-tls" not found Jul 08 19:12:15.612595 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.612563 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc"] Jul 08 19:12:15.637881 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.637855 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4"] Jul 08 19:12:15.638051 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.637984 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc" Jul 08 19:12:15.640706 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.640686 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 19:12:15.641263 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.641242 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mfnl2\"" Jul 08 19:12:15.641369 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.641241 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:12:15.662541 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.662518 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-54f56f5b6f-8l27s"] Jul 08 19:12:15.662641 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.662626 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:15.665141 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.665122 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 19:12:15.665141 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.665137 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:12:15.665275 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.665257 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nv5mm\"" Jul 08 19:12:15.665503 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.665477 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 19:12:15.686532 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.686512 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc"] Jul 08 19:12:15.686641 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.686545 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4"] Jul 08 19:12:15.686641 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.686570 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54f56f5b6f-8l27s"] Jul 08 19:12:15.686709 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.686668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.689370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.689319 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 19:12:15.689677 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.689658 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 19:12:15.689769 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.689718 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f7crt\"" Jul 08 19:12:15.689769 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.689725 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 19:12:15.689769 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.689728 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 19:12:15.689949 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.689728 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 19:12:15.689949 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.689858 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 19:12:15.701270 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.701250 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwd6\" (UniqueName: \"kubernetes.io/projected/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-kube-api-access-2wwd6\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:15.701377 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.701280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.701377 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.701298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5t9\" (UniqueName: \"kubernetes.io/projected/94079af4-8a24-474d-871d-8bf792b247c4-kube-api-access-jb5t9\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.701377 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.701317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-default-certificate\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.701377 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.701339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-stats-auth\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.701527 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.701454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lg4s\" (UniqueName: \"kubernetes.io/projected/a82840dc-4271-44e5-bd09-1092c7de4858-kube-api-access-6lg4s\") pod \"volume-data-source-validator-5d456fdbcd-gd8tc\" (UID: \"a82840dc-4271-44e5-bd09-1092c7de4858\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc" Jul 08 19:12:15.701527 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.701489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.701527 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.701509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:15.802052 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.802016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.802052 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.802054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:15.802324 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.802112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwd6\" (UniqueName: \"kubernetes.io/projected/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-kube-api-access-2wwd6\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:15.802324 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.802135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.802324 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.802155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5t9\" (UniqueName: \"kubernetes.io/projected/94079af4-8a24-474d-871d-8bf792b247c4-kube-api-access-jb5t9\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.802324 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:15.802192 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:12:15.802324 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:15.802276 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:16.302255095 +0000 UTC m=+53.604559259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : secret "router-metrics-certs-default" not found Jul 08 19:12:15.802324 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:15.802284 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:12:15.802324 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:15.802299 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:16.302288283 +0000 UTC m=+53.604592432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : configmap references non-existent config key: service-ca.crt Jul 08 19:12:15.802324 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.802328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-default-certificate\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.802797 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:15.802339 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls podName:59b16ab0-1fea-4f92-8bd6-76ac743eb3bf nodeName:}" failed. No retries permitted until 2026-07-08 19:12:16.302322859 +0000 UTC m=+53.604627025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-zp6z4" (UID: "59b16ab0-1fea-4f92-8bd6-76ac743eb3bf") : secret "samples-operator-tls" not found Jul 08 19:12:15.802797 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.802395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-stats-auth\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.802797 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.802459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lg4s\" (UniqueName: \"kubernetes.io/projected/a82840dc-4271-44e5-bd09-1092c7de4858-kube-api-access-6lg4s\") pod \"volume-data-source-validator-5d456fdbcd-gd8tc\" (UID: \"a82840dc-4271-44e5-bd09-1092c7de4858\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc" Jul 08 19:12:15.805849 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.805828 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-stats-auth\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.805945 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.805834 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-default-certificate\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.813059 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.813034 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5t9\" (UniqueName: \"kubernetes.io/projected/94079af4-8a24-474d-871d-8bf792b247c4-kube-api-access-jb5t9\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:15.813151 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.813057 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lg4s\" (UniqueName: \"kubernetes.io/projected/a82840dc-4271-44e5-bd09-1092c7de4858-kube-api-access-6lg4s\") pod \"volume-data-source-validator-5d456fdbcd-gd8tc\" (UID: \"a82840dc-4271-44e5-bd09-1092c7de4858\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc" Jul 08 19:12:15.813379 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.813339 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwd6\" (UniqueName: \"kubernetes.io/projected/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-kube-api-access-2wwd6\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:15.946894 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:15.946800 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc" Jul 08 19:12:16.092918 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:16.092885 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc"] Jul 08 19:12:16.096271 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:12:16.096248 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82840dc_4271_44e5_bd09_1092c7de4858.slice/crio-61460b080b09ad02a9a1f3fbc5ac786260b63328936cd3a3ecdee9fd6146d754 WatchSource:0}: Error finding container 61460b080b09ad02a9a1f3fbc5ac786260b63328936cd3a3ecdee9fd6146d754: Status 404 returned error can't find the container with id 61460b080b09ad02a9a1f3fbc5ac786260b63328936cd3a3ecdee9fd6146d754 Jul 08 19:12:16.307564 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:16.307537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:16.307726 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:16.307702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:16.307726 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:16.307715 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:17.307698054 +0000 UTC m=+54.610002203 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : configmap references non-existent config key: service-ca.crt Jul 08 19:12:16.307830 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:16.307750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:16.307830 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:16.307825 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:12:16.307906 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:16.307865 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:12:16.307906 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:16.307868 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:17.307857291 +0000 UTC m=+54.610161436 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : secret "router-metrics-certs-default" not found Jul 08 19:12:16.307906 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:16.307904 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls podName:59b16ab0-1fea-4f92-8bd6-76ac743eb3bf nodeName:}" failed. No retries permitted until 2026-07-08 19:12:17.307895239 +0000 UTC m=+54.610199383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-zp6z4" (UID: "59b16ab0-1fea-4f92-8bd6-76ac743eb3bf") : secret "samples-operator-tls" not found Jul 08 19:12:16.524331 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:16.524293 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc" event={"ID":"a82840dc-4271-44e5-bd09-1092c7de4858","Type":"ContainerStarted","Data":"61460b080b09ad02a9a1f3fbc5ac786260b63328936cd3a3ecdee9fd6146d754"} Jul 08 19:12:17.316101 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:17.316069 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:17.316567 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:17.316138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:17.316567 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:17.316159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:17.316567 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:17.316249 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:12:17.316567 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:17.316260 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:12:17.316567 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:17.316291 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:19.31626859 +0000 UTC m=+56.618572740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : configmap references non-existent config key: service-ca.crt Jul 08 19:12:17.316567 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:17.316315 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:19.31630397 +0000 UTC m=+56.618608119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : secret "router-metrics-certs-default" not found Jul 08 19:12:17.316567 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:17.316328 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls podName:59b16ab0-1fea-4f92-8bd6-76ac743eb3bf nodeName:}" failed. No retries permitted until 2026-07-08 19:12:19.316322158 +0000 UTC m=+56.618626302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-zp6z4" (UID: "59b16ab0-1fea-4f92-8bd6-76ac743eb3bf") : secret "samples-operator-tls" not found Jul 08 19:12:18.531418 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:18.531387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc" event={"ID":"a82840dc-4271-44e5-bd09-1092c7de4858","Type":"ContainerStarted","Data":"337d835a5e9ceec5d0c59e4f585a131f7282da920be8e0a2231ecd17fcfac74b"} Jul 08 19:12:18.548031 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:18.547990 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gd8tc" podStartSLOduration=1.66873271 podStartE2EDuration="3.547976138s" totalImagesPullingTime="1.879243428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:18.546433107 +0000 UTC m=+55.848737297" watchObservedRunningTime="2026-07-08 19:12:18.547976138 +0000 UTC m=+55.850280305" Jul 08 19:12:19.333219 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:19.333170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:19.333436 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:19.333262 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:19.333436 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:19.333283 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:19.333436 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:19.333359 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:23.333326125 +0000 UTC m=+60.635630270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : configmap references non-existent config key: service-ca.crt Jul 08 19:12:19.333593 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:19.333444 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:12:19.333593 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:19.333501 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:23.333483573 +0000 UTC m=+60.635787737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : secret "router-metrics-certs-default" not found Jul 08 19:12:19.333593 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:19.333529 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:12:19.333593 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:19.333564 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls podName:59b16ab0-1fea-4f92-8bd6-76ac743eb3bf nodeName:}" failed. No retries permitted until 2026-07-08 19:12:23.3335547 +0000 UTC m=+60.635858845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-zp6z4" (UID: "59b16ab0-1fea-4f92-8bd6-76ac743eb3bf") : secret "samples-operator-tls" not found Jul 08 19:12:22.934334 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.934299 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m4752"] Jul 08 19:12:22.937126 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.937104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:22.941091 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.941069 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 19:12:22.941882 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.941861 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 19:12:22.941958 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.941866 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 19:12:22.941958 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.941867 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qc64c\"" Jul 08 19:12:22.941958 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.941866 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 19:12:22.959637 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.959607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-data-volume\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:22.959779 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.959653 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:22.959841 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.959818 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbdxk\" (UniqueName: \"kubernetes.io/projected/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-api-access-nbdxk\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:22.959892 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.959861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-crio-socket\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:22.959944 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.959893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:22.967052 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:22.967032 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m4752"] Jul 08 19:12:23.060832 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.060791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:23.060832 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.060837 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-data-volume\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:23.061059 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.060860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:23.061100 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.061055 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbdxk\" (UniqueName: \"kubernetes.io/projected/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-api-access-nbdxk\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:23.061100 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.061094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-crio-socket\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:23.061182 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.061168 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-crio-socket\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:23.061219 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.061182 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-data-volume\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:23.063892 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.063873 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 19:12:23.063945 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.063879 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 19:12:23.071200 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.071185 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:12:23.071256 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.071189 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:12:23.071288 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.071235 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm podName:17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:23.571220195 +0000 UTC m=+60.873524343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4752" (UID: "17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57") : configmap "kube-rbac-proxy" not found Jul 08 19:12:23.071337 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.071295 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls podName:17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:23.571281247 +0000 UTC m=+60.873585395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4752" (UID: "17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57") : secret "insights-runtime-extractor-tls" not found Jul 08 19:12:23.081982 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.081962 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 19:12:23.094525 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.094503 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 19:12:23.103060 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.103026 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbdxk\" (UniqueName: \"kubernetes.io/projected/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-api-access-nbdxk\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:23.108693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.108672 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m"] Jul 08 19:12:23.110997 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.110979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m" Jul 08 19:12:23.113896 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.113876 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 08 19:12:23.113981 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.113909 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 08 19:12:23.114023 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.113992 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-br2lz\"" Jul 08 19:12:23.124929 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.124906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m"] Jul 08 19:12:23.161505 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.161480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq99n\" (UniqueName: \"kubernetes.io/projected/985ee924-94f3-4673-b725-16a6d648e917-kube-api-access-cq99n\") pod \"migrator-d57d58cc9-6ph8m\" (UID: \"985ee924-94f3-4673-b725-16a6d648e917\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m" Jul 08 19:12:23.262004 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.261920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq99n\" (UniqueName: \"kubernetes.io/projected/985ee924-94f3-4673-b725-16a6d648e917-kube-api-access-cq99n\") pod \"migrator-d57d58cc9-6ph8m\" (UID: \"985ee924-94f3-4673-b725-16a6d648e917\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m" Jul 08 19:12:23.271381 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.271338 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq99n\" (UniqueName: \"kubernetes.io/projected/985ee924-94f3-4673-b725-16a6d648e917-kube-api-access-cq99n\") pod \"migrator-d57d58cc9-6ph8m\" (UID: \"985ee924-94f3-4673-b725-16a6d648e917\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m" Jul 08 19:12:23.363143 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.363110 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:23.363306 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.363151 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:23.363306 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.363266 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:31.363252902 +0000 UTC m=+68.665557047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : configmap references non-existent config key: service-ca.crt Jul 08 19:12:23.363306 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.363266 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:12:23.363306 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.363287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:23.363491 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.363326 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls podName:59b16ab0-1fea-4f92-8bd6-76ac743eb3bf nodeName:}" failed. No retries permitted until 2026-07-08 19:12:31.36330896 +0000 UTC m=+68.665613111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-zp6z4" (UID: "59b16ab0-1fea-4f92-8bd6-76ac743eb3bf") : secret "samples-operator-tls" not found Jul 08 19:12:23.363491 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.363377 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:12:23.363865 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.363841 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:31.363403267 +0000 UTC m=+68.665707428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : secret "router-metrics-certs-default" not found Jul 08 19:12:23.419609 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.419586 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m" Jul 08 19:12:23.538105 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.538080 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m"] Jul 08 19:12:23.540728 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:12:23.540706 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985ee924_94f3_4673_b725_16a6d648e917.slice/crio-d909746498866190803fc4f9ff51f4080ca2b2b3b04300b14d858f4e1603c248 WatchSource:0}: Error finding container d909746498866190803fc4f9ff51f4080ca2b2b3b04300b14d858f4e1603c248: Status 404 returned error can't find the container with id d909746498866190803fc4f9ff51f4080ca2b2b3b04300b14d858f4e1603c248 Jul 08 19:12:23.545238 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.545213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m" event={"ID":"985ee924-94f3-4673-b725-16a6d648e917","Type":"ContainerStarted","Data":"d909746498866190803fc4f9ff51f4080ca2b2b3b04300b14d858f4e1603c248"} Jul 08 19:12:23.665669 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.665634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:23.665838 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.665754 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:12:23.665838 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:23.665764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:23.665925 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.665869 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm podName:17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:24.665853509 +0000 UTC m=+61.968157654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4752" (UID: "17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57") : configmap "kube-rbac-proxy" not found Jul 08 19:12:23.665925 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.665877 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:12:23.665925 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:23.665921 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls podName:17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:24.665910141 +0000 UTC m=+61.968214285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4752" (UID: "17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57") : secret "insights-runtime-extractor-tls" not found Jul 08 19:12:24.461211 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:24.461184 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hr5zg" Jul 08 19:12:24.672739 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:24.672706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:24.672739 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:24.672757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:24.673066 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:24.672948 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:12:24.673066 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:24.672971 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:12:24.673066 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:24.673018 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls podName:17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:26.673000105 +0000 UTC m=+63.975304250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4752" (UID: "17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57") : secret "insights-runtime-extractor-tls" not found Jul 08 19:12:24.673066 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:24.673039 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm podName:17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:26.673029008 +0000 UTC m=+63.975333155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4752" (UID: "17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57") : configmap "kube-rbac-proxy" not found Jul 08 19:12:24.724121 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:24.724047 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zzbhq_1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca/dns-node-resolver/0.log" Jul 08 19:12:25.527469 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:25.527448 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lldql_d385dcd3-5579-42a7-af3d-c9f272c9a568/node-ca/0.log" Jul 08 19:12:25.551517 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:25.551490 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m" event={"ID":"985ee924-94f3-4673-b725-16a6d648e917","Type":"ContainerStarted","Data":"4073d1a1b6146a482e9cf6cc9e6c23990b388b288c4e9a1e377af1896b6324b8"} Jul 08 19:12:25.551664 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:25.551545 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m" event={"ID":"985ee924-94f3-4673-b725-16a6d648e917","Type":"ContainerStarted","Data":"b3e7a0d8d722020ab5a0b120eb1da85a1c82a5b514f016c22bdc1cfff6128367"} Jul 08 19:12:25.570711 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:25.570661 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-6ph8m" podStartSLOduration=0.744878215 podStartE2EDuration="2.570645282s" totalImagesPullingTime="1.825767067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:25.570254272 +0000 UTC m=+62.872558440" watchObservedRunningTime="2026-07-08 19:12:25.570645282 +0000 UTC m=+62.872949492" Jul 08 19:12:26.325572 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:26.325540 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pbwkx_3db481d0-9cdf-46be-a738-505e054ba95b/serve-healthcheck-canary/0.log" Jul 08 19:12:26.690677 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:26.690598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:26.691037 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:26.690723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:26.691037 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:26.690725 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:12:26.691037 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:26.690784 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:12:26.691037 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:26.690795 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm podName:17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:30.690781681 +0000 UTC m=+67.993085826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4752" (UID: "17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57") : configmap "kube-rbac-proxy" not found Jul 08 19:12:26.691037 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:26.690833 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls podName:17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:30.690820487 +0000 UTC m=+67.993124632 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4752" (UID: "17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57") : secret "insights-runtime-extractor-tls" not found Jul 08 19:12:27.903272 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:27.903218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:12:27.907369 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:27.907330 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 19:12:27.916503 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:27.916483 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b982417-f3a9-43ff-9a76-920bd553524e-original-pull-secret\") pod \"global-pull-secret-syncer-6qs6k\" (UID: \"5b982417-f3a9-43ff-9a76-920bd553524e\") " pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:12:28.004420 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.004388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssw8\" (UniqueName: \"kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8\") pod \"network-check-target-b27qk\" (UID: \"429a71ea-4e0e-4573-8394-bcbe65957d13\") " pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:12:28.004576 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.004441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:12:28.007672 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.007651 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 19:12:28.007753 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.007721 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 19:12:28.015395 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:28.015375 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 19:12:28.015454 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:28.015443 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs podName:8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6 nodeName:}" failed. No retries permitted until 2026-07-08 19:13:32.01542249 +0000 UTC m=+129.317726648 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs") pod "network-metrics-daemon-nfnnj" (UID: "8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6") : secret "metrics-daemon-secret" not found Jul 08 19:12:28.018267 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.018251 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 19:12:28.028087 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.028069 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ssw8\" (UniqueName: \"kubernetes.io/projected/429a71ea-4e0e-4573-8394-bcbe65957d13-kube-api-access-6ssw8\") pod \"network-check-target-b27qk\" (UID: \"429a71ea-4e0e-4573-8394-bcbe65957d13\") " pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:12:28.174370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.174278 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-92l7d\"" Jul 08 19:12:28.175359 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.175327 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qs6k" Jul 08 19:12:28.182045 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.182024 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:12:28.305750 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.305726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6qs6k"] Jul 08 19:12:28.308017 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:12:28.307993 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b982417_f3a9_43ff_9a76_920bd553524e.slice/crio-e47873503d6cc7c6f773723c981bb3cef611f5cd38e4de2f49ae770bec317880 WatchSource:0}: Error finding container e47873503d6cc7c6f773723c981bb3cef611f5cd38e4de2f49ae770bec317880: Status 404 returned error can't find the container with id e47873503d6cc7c6f773723c981bb3cef611f5cd38e4de2f49ae770bec317880 Jul 08 19:12:28.322410 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.322326 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b27qk"] Jul 08 19:12:28.324466 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:12:28.324441 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429a71ea_4e0e_4573_8394_bcbe65957d13.slice/crio-1eb3e4037084693c0cff6da2ec2725ed64948340447227c4a7e66289df89ad53 WatchSource:0}: Error finding container 1eb3e4037084693c0cff6da2ec2725ed64948340447227c4a7e66289df89ad53: Status 404 returned error can't find the container with id 1eb3e4037084693c0cff6da2ec2725ed64948340447227c4a7e66289df89ad53 Jul 08 19:12:28.560979 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.560946 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b27qk" event={"ID":"429a71ea-4e0e-4573-8394-bcbe65957d13","Type":"ContainerStarted","Data":"1eb3e4037084693c0cff6da2ec2725ed64948340447227c4a7e66289df89ad53"} Jul 08 19:12:28.562043 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:28.562021 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6qs6k" event={"ID":"5b982417-f3a9-43ff-9a76-920bd553524e","Type":"ContainerStarted","Data":"e47873503d6cc7c6f773723c981bb3cef611f5cd38e4de2f49ae770bec317880"} Jul 08 19:12:30.427229 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:30.427189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:12:30.427742 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:30.427246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:12:30.427742 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:30.427381 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 19:12:30.427742 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:30.427405 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ff55887b9-vn4sp: secret "image-registry-tls" not found Jul 08 19:12:30.427742 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:30.427472 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls podName:1cad1733-48a4-4a9b-bd22-fb2e7b35c008 nodeName:}" failed. No retries permitted until 2026-07-08 19:13:02.427449419 +0000 UTC m=+99.729753580 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls") pod "image-registry-5ff55887b9-vn4sp" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008") : secret "image-registry-tls" not found Jul 08 19:12:30.430240 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:30.430191 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/53d10eff-78be-422c-b7ce-13d970eca76d-metrics-tls\") pod \"dns-default-8njxt\" (UID: \"53d10eff-78be-422c-b7ce-13d970eca76d\") " pod="openshift-dns/dns-default-8njxt" Jul 08 19:12:30.630804 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:30.630771 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4xh72\"" Jul 08 19:12:30.638599 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:30.638577 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8njxt" Jul 08 19:12:30.730188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:30.730105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:30.730188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:30.730157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:30.730430 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:30.730257 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:12:30.730430 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:30.730270 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:12:30.730430 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:30.730321 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm podName:17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:38.730302985 +0000 UTC m=+76.032607144 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4752" (UID: "17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57") : configmap "kube-rbac-proxy" not found Jul 08 19:12:30.730430 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:30.730359 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls podName:17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:38.730331443 +0000 UTC m=+76.032635589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4752" (UID: "17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57") : secret "insights-runtime-extractor-tls" not found Jul 08 19:12:31.283657 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.283618 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8njxt"] Jul 08 19:12:31.286999 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:12:31.286974 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d10eff_78be_422c_b7ce_13d970eca76d.slice/crio-a48b099ddc43883ce63c8de09025b029f456ed4b6508c98980a0c1f17268ff7c WatchSource:0}: Error finding container a48b099ddc43883ce63c8de09025b029f456ed4b6508c98980a0c1f17268ff7c: Status 404 returned error can't find the container with id a48b099ddc43883ce63c8de09025b029f456ed4b6508c98980a0c1f17268ff7c Jul 08 19:12:31.435626 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.435524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:31.435626 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.435572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:31.436076 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:31.435698 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:47.435681087 +0000 UTC m=+84.737985248 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : configmap references non-existent config key: service-ca.crt Jul 08 19:12:31.436076 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.435727 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:31.436076 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:31.435816 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:12:31.436076 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:12:31.435840 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs podName:94079af4-8a24-474d-871d-8bf792b247c4 nodeName:}" failed. No retries permitted until 2026-07-08 19:12:47.435833782 +0000 UTC m=+84.738137926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs") pod "router-default-54f56f5b6f-8l27s" (UID: "94079af4-8a24-474d-871d-8bf792b247c4") : secret "router-metrics-certs-default" not found Jul 08 19:12:31.438002 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.437978 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/59b16ab0-1fea-4f92-8bd6-76ac743eb3bf-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-zp6z4\" (UID: \"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:31.572568 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.572535 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nv5mm\"" Jul 08 19:12:31.574238 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.574206 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8njxt" event={"ID":"53d10eff-78be-422c-b7ce-13d970eca76d","Type":"ContainerStarted","Data":"a48b099ddc43883ce63c8de09025b029f456ed4b6508c98980a0c1f17268ff7c"} Jul 08 19:12:31.575897 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.575869 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b27qk" event={"ID":"429a71ea-4e0e-4573-8394-bcbe65957d13","Type":"ContainerStarted","Data":"56d4f3fd2081317a70ce00b88682a2ade5d6093051f26aaa88c0697cbf1d467e"} Jul 08 19:12:31.576007 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.575975 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:12:31.580466 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.580449 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" Jul 08 19:12:31.593424 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.593375 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b27qk" podStartSLOduration=65.726841124 podStartE2EDuration="1m8.593338054s" totalImagesPullingTime="2.86649693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:31.590956019 +0000 UTC m=+68.893260187" watchObservedRunningTime="2026-07-08 19:12:31.593338054 +0000 UTC m=+68.895642222" Jul 08 19:12:31.706514 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:31.706416 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4"] Jul 08 19:12:32.583428 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:32.583384 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" event={"ID":"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf","Type":"ContainerStarted","Data":"c6e254850d7b0cf544bbfa73342169384070d9db15437896e9b34052a025f914"} Jul 08 19:12:35.595322 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:35.595294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" event={"ID":"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf","Type":"ContainerStarted","Data":"3f8cc51c3b2cea658661f6c5325576afd27874353f65bd4a857c09fcb1a719ec"} Jul 08 19:12:36.601492 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:36.601458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6qs6k" event={"ID":"5b982417-f3a9-43ff-9a76-920bd553524e","Type":"ContainerStarted","Data":"d10ad738d369674f3904ec52396c5f1569f0dfd3d0ac0bef7101f8b8e637ab7c"} Jul 08 19:12:36.603055 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:36.603024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8njxt" event={"ID":"53d10eff-78be-422c-b7ce-13d970eca76d","Type":"ContainerStarted","Data":"2ed63746f7b02a5164b8d07dac05f6bf9da3c898be8051f5773f44ba1deddcb6"} Jul 08 19:12:36.603188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:36.603081 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8njxt" event={"ID":"53d10eff-78be-422c-b7ce-13d970eca76d","Type":"ContainerStarted","Data":"4cac7bb1f273561a80cd378bc63b92995a1eecc7d253be4929e4771a2323e052"} Jul 08 19:12:36.603188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:36.603111 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8njxt" Jul 08 19:12:36.604830 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:36.604809 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" event={"ID":"59b16ab0-1fea-4f92-8bd6-76ac743eb3bf","Type":"ContainerStarted","Data":"8d76e528bdb152f95bc1da27f70fcf8f0275cff32db9b7d30bd5956e8f5e4114"} Jul 08 19:12:36.632006 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:36.631967 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6qs6k" podStartSLOduration=65.428948414 podStartE2EDuration="1m12.631953824s" totalImagesPullingTime="7.20300541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:36.630910398 +0000 UTC m=+73.933214578" watchObservedRunningTime="2026-07-08 19:12:36.631953824 +0000 UTC m=+73.934258001" Jul 08 19:12:36.654674 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:36.654626 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-zp6z4" podStartSLOduration=17.882883858 podStartE2EDuration="21.654612278s" totalImagesPullingTime="3.77172842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:36.653721092 +0000 UTC m=+73.956025259" watchObservedRunningTime="2026-07-08 19:12:36.654612278 +0000 UTC m=+73.956916445" Jul 08 19:12:36.671738 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:36.671703 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8njxt" podStartSLOduration=34.445389073 podStartE2EDuration="38.671689956s" totalImagesPullingTime="4.226300883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:36.670492943 +0000 UTC m=+73.972797111" watchObservedRunningTime="2026-07-08 19:12:36.671689956 +0000 UTC m=+73.973994122" Jul 08 19:12:38.799888 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:38.799847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:38.800277 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:38.799897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:38.800408 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:38.800391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:38.802159 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:38.802137 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4752\" (UID: \"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57\") " pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:38.848869 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:38.848839 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qc64c\"" Jul 08 19:12:38.856332 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:38.856310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4752" Jul 08 19:12:38.998840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:38.998807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m4752"] Jul 08 19:12:39.001641 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:12:39.001614 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e39ded_a2a4_4c7e_a350_6ca7fe8b3d57.slice/crio-41e5cc65d408dde443ac55fb375ead3463ab2ccb1548755387d783537a56331a WatchSource:0}: Error finding container 41e5cc65d408dde443ac55fb375ead3463ab2ccb1548755387d783537a56331a: Status 404 returned error can't find the container with id 41e5cc65d408dde443ac55fb375ead3463ab2ccb1548755387d783537a56331a Jul 08 19:12:39.617000 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:39.616960 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4752" event={"ID":"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57","Type":"ContainerStarted","Data":"41e5cc65d408dde443ac55fb375ead3463ab2ccb1548755387d783537a56331a"} Jul 08 19:12:43.634009 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:43.633972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4752" event={"ID":"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57","Type":"ContainerStarted","Data":"11fca67eb6cf1215737be43b0513126bc0e7f1e36a7d81895388287d9a3347cd"} Jul 08 19:12:43.634419 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:43.634037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4752" event={"ID":"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57","Type":"ContainerStarted","Data":"5a7484b75b8da78c7a54dfe3dcecdc7c996b1f4345097c3063ff050847fb9215"} Jul 08 19:12:46.610785 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:46.610756 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8njxt" Jul 08 19:12:46.646683 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:46.646656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4752" event={"ID":"17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57","Type":"ContainerStarted","Data":"36448d19428cf9933b7f577700b55d1c66748a311f89fb846769bc04ebeb4bdc"} Jul 08 19:12:46.665120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:46.665072 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m4752" podStartSLOduration=17.796398071 podStartE2EDuration="24.665059203s" totalImagesPullingTime="6.868661132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:12:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:46.66449674 +0000 UTC m=+83.966800908" watchObservedRunningTime="2026-07-08 19:12:46.665059203 +0000 UTC m=+83.967363369" Jul 08 19:12:47.473920 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:47.473862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:47.473920 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:47.473926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:47.474594 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:47.474569 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94079af4-8a24-474d-871d-8bf792b247c4-service-ca-bundle\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:47.476190 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:47.476173 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94079af4-8a24-474d-871d-8bf792b247c4-metrics-certs\") pod \"router-default-54f56f5b6f-8l27s\" (UID: \"94079af4-8a24-474d-871d-8bf792b247c4\") " pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:47.497933 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:47.497911 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f7crt\"" Jul 08 19:12:47.505332 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:47.505305 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:47.647724 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:47.647697 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54f56f5b6f-8l27s"] Jul 08 19:12:47.650675 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:12:47.650645 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94079af4_8a24_474d_871d_8bf792b247c4.slice/crio-c38c6ffa50af35197bfb752bf8857c23ec62968159dbb6050ff93a95fb7fde83 WatchSource:0}: Error finding container c38c6ffa50af35197bfb752bf8857c23ec62968159dbb6050ff93a95fb7fde83: Status 404 returned error can't find the container with id c38c6ffa50af35197bfb752bf8857c23ec62968159dbb6050ff93a95fb7fde83 Jul 08 19:12:48.613013 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.612978 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g"] Jul 08 19:12:48.616108 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.616091 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.619635 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.619612 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 19:12:48.621214 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.621193 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 08 19:12:48.621330 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.621243 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 19:12:48.621474 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.621446 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 19:12:48.622474 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.622324 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52"] Jul 08 19:12:48.625283 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.625266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" Jul 08 19:12:48.628308 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.628291 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 08 19:12:48.629765 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.629746 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mdcm6\"" Jul 08 19:12:48.635803 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.635784 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g"] Jul 08 19:12:48.636613 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.636596 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52"] Jul 08 19:12:48.655311 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.655273 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54f56f5b6f-8l27s" event={"ID":"94079af4-8a24-474d-871d-8bf792b247c4","Type":"ContainerStarted","Data":"395cebf3a517b406ddbbdadf6d6cabc6080209aef8c91ba0f20779a045da59c3"} Jul 08 19:12:48.655726 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.655323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54f56f5b6f-8l27s" event={"ID":"94079af4-8a24-474d-871d-8bf792b247c4","Type":"ContainerStarted","Data":"c38c6ffa50af35197bfb752bf8857c23ec62968159dbb6050ff93a95fb7fde83"} Jul 08 19:12:48.700156 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.700110 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-54f56f5b6f-8l27s" podStartSLOduration=33.700094194 podStartE2EDuration="33.700094194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:48.699744719 +0000 UTC m=+86.002048886" watchObservedRunningTime="2026-07-08 19:12:48.700094194 +0000 UTC m=+86.002398361" Jul 08 19:12:48.716013 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.714185 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-lljqn"] Jul 08 19:12:48.718215 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.718194 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-lljqn" Jul 08 19:12:48.724600 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.724577 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-cfkxl\"" Jul 08 19:12:48.725036 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.725024 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 19:12:48.725088 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.725047 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 19:12:48.732578 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.732559 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-lljqn"] Jul 08 19:12:48.786557 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.786524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7qk6\" (UniqueName: \"kubernetes.io/projected/c3ad5067-d02e-4162-b82c-a7028d22e844-kube-api-access-m7qk6\") pod \"managed-serviceaccount-addon-agent-84d44b889b-c6r52\" (UID: \"c3ad5067-d02e-4162-b82c-a7028d22e844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" Jul 08 19:12:48.786721 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.786643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7r6\" (UniqueName: \"kubernetes.io/projected/32905ee7-0c58-41b5-99ff-e42f59fc1039-kube-api-access-gv7r6\") pod \"klusterlet-addon-workmgr-784684cd49-29d9g\" (UID: \"32905ee7-0c58-41b5-99ff-e42f59fc1039\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.786721 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.786695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32905ee7-0c58-41b5-99ff-e42f59fc1039-tmp\") pod \"klusterlet-addon-workmgr-784684cd49-29d9g\" (UID: \"32905ee7-0c58-41b5-99ff-e42f59fc1039\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.786825 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.786725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/32905ee7-0c58-41b5-99ff-e42f59fc1039-klusterlet-config\") pod \"klusterlet-addon-workmgr-784684cd49-29d9g\" (UID: \"32905ee7-0c58-41b5-99ff-e42f59fc1039\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.786933 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.786815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3ad5067-d02e-4162-b82c-a7028d22e844-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84d44b889b-c6r52\" (UID: \"c3ad5067-d02e-4162-b82c-a7028d22e844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" Jul 08 19:12:48.887754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.887671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7klrp\" (UniqueName: \"kubernetes.io/projected/27d50fcc-7031-4354-8db6-023f93cad601-kube-api-access-7klrp\") pod \"downloads-685bfb5f64-lljqn\" (UID: \"27d50fcc-7031-4354-8db6-023f93cad601\") " pod="openshift-console/downloads-685bfb5f64-lljqn" Jul 08 19:12:48.887754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.887725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7r6\" (UniqueName: \"kubernetes.io/projected/32905ee7-0c58-41b5-99ff-e42f59fc1039-kube-api-access-gv7r6\") pod \"klusterlet-addon-workmgr-784684cd49-29d9g\" (UID: \"32905ee7-0c58-41b5-99ff-e42f59fc1039\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.887954 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.887782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32905ee7-0c58-41b5-99ff-e42f59fc1039-tmp\") pod \"klusterlet-addon-workmgr-784684cd49-29d9g\" (UID: \"32905ee7-0c58-41b5-99ff-e42f59fc1039\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.887954 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.887832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/32905ee7-0c58-41b5-99ff-e42f59fc1039-klusterlet-config\") pod \"klusterlet-addon-workmgr-784684cd49-29d9g\" (UID: \"32905ee7-0c58-41b5-99ff-e42f59fc1039\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.887954 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.887899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3ad5067-d02e-4162-b82c-a7028d22e844-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84d44b889b-c6r52\" (UID: \"c3ad5067-d02e-4162-b82c-a7028d22e844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" Jul 08 19:12:48.887954 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.887928 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7qk6\" (UniqueName: \"kubernetes.io/projected/c3ad5067-d02e-4162-b82c-a7028d22e844-kube-api-access-m7qk6\") pod \"managed-serviceaccount-addon-agent-84d44b889b-c6r52\" (UID: \"c3ad5067-d02e-4162-b82c-a7028d22e844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" Jul 08 19:12:48.888312 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.888163 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32905ee7-0c58-41b5-99ff-e42f59fc1039-tmp\") pod \"klusterlet-addon-workmgr-784684cd49-29d9g\" (UID: \"32905ee7-0c58-41b5-99ff-e42f59fc1039\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.890184 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.890163 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/32905ee7-0c58-41b5-99ff-e42f59fc1039-klusterlet-config\") pod \"klusterlet-addon-workmgr-784684cd49-29d9g\" (UID: \"32905ee7-0c58-41b5-99ff-e42f59fc1039\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.890294 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.890210 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3ad5067-d02e-4162-b82c-a7028d22e844-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84d44b889b-c6r52\" (UID: \"c3ad5067-d02e-4162-b82c-a7028d22e844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" Jul 08 19:12:48.900718 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.900690 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7r6\" (UniqueName: \"kubernetes.io/projected/32905ee7-0c58-41b5-99ff-e42f59fc1039-kube-api-access-gv7r6\") pod \"klusterlet-addon-workmgr-784684cd49-29d9g\" (UID: \"32905ee7-0c58-41b5-99ff-e42f59fc1039\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.901190 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.901172 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7qk6\" (UniqueName: \"kubernetes.io/projected/c3ad5067-d02e-4162-b82c-a7028d22e844-kube-api-access-m7qk6\") pod \"managed-serviceaccount-addon-agent-84d44b889b-c6r52\" (UID: \"c3ad5067-d02e-4162-b82c-a7028d22e844\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" Jul 08 19:12:48.925249 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.925228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:48.942880 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.942861 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" Jul 08 19:12:48.988672 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:48.988642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7klrp\" (UniqueName: \"kubernetes.io/projected/27d50fcc-7031-4354-8db6-023f93cad601-kube-api-access-7klrp\") pod \"downloads-685bfb5f64-lljqn\" (UID: \"27d50fcc-7031-4354-8db6-023f93cad601\") " pod="openshift-console/downloads-685bfb5f64-lljqn" Jul 08 19:12:49.007959 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.007920 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7klrp\" (UniqueName: \"kubernetes.io/projected/27d50fcc-7031-4354-8db6-023f93cad601-kube-api-access-7klrp\") pod \"downloads-685bfb5f64-lljqn\" (UID: \"27d50fcc-7031-4354-8db6-023f93cad601\") " pod="openshift-console/downloads-685bfb5f64-lljqn" Jul 08 19:12:49.026653 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.026540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-lljqn" Jul 08 19:12:49.067964 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.067940 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g"] Jul 08 19:12:49.070824 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:12:49.070798 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32905ee7_0c58_41b5_99ff_e42f59fc1039.slice/crio-6c083d3c23dfbab6a5373edf888caae070da2731f46ad4dea45442c40691cee7 WatchSource:0}: Error finding container 6c083d3c23dfbab6a5373edf888caae070da2731f46ad4dea45442c40691cee7: Status 404 returned error can't find the container with id 6c083d3c23dfbab6a5373edf888caae070da2731f46ad4dea45442c40691cee7 Jul 08 19:12:49.085019 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.084987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52"] Jul 08 19:12:49.088440 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:12:49.088410 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3ad5067_d02e_4162_b82c_a7028d22e844.slice/crio-55e1616067b05740a09c87984a7824eb1daf52b0709013673c8a410abd557f9e WatchSource:0}: Error finding container 55e1616067b05740a09c87984a7824eb1daf52b0709013673c8a410abd557f9e: Status 404 returned error can't find the container with id 55e1616067b05740a09c87984a7824eb1daf52b0709013673c8a410abd557f9e Jul 08 19:12:49.175519 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.175498 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-lljqn"] Jul 08 19:12:49.177811 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:12:49.177785 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d50fcc_7031_4354_8db6_023f93cad601.slice/crio-d6648a76cce4094f93e6a1ab246b8f43c7cb3d8798e3bbb662cba212f185f08d WatchSource:0}: Error finding container d6648a76cce4094f93e6a1ab246b8f43c7cb3d8798e3bbb662cba212f185f08d: Status 404 returned error can't find the container with id d6648a76cce4094f93e6a1ab246b8f43c7cb3d8798e3bbb662cba212f185f08d Jul 08 19:12:49.505559 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.505470 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:49.507923 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.507900 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:49.659410 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.659377 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-lljqn" event={"ID":"27d50fcc-7031-4354-8db6-023f93cad601","Type":"ContainerStarted","Data":"d6648a76cce4094f93e6a1ab246b8f43c7cb3d8798e3bbb662cba212f185f08d"} Jul 08 19:12:49.660403 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.660373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" event={"ID":"c3ad5067-d02e-4162-b82c-a7028d22e844","Type":"ContainerStarted","Data":"55e1616067b05740a09c87984a7824eb1daf52b0709013673c8a410abd557f9e"} Jul 08 19:12:49.661468 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.661445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" event={"ID":"32905ee7-0c58-41b5-99ff-e42f59fc1039","Type":"ContainerStarted","Data":"6c083d3c23dfbab6a5373edf888caae070da2731f46ad4dea45442c40691cee7"} Jul 08 19:12:49.661707 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.661692 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:49.663199 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:49.663181 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-54f56f5b6f-8l27s" Jul 08 19:12:52.677402 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:52.677325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" event={"ID":"c3ad5067-d02e-4162-b82c-a7028d22e844","Type":"ContainerStarted","Data":"aa58023af431ddb4618f271f863161c12de75a75727fb53351078b5046a7fe21"} Jul 08 19:12:52.695706 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:52.695639 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84d44b889b-c6r52" podStartSLOduration=1.6633882949999999 podStartE2EDuration="4.695614842s" totalImagesPullingTime="3.032226547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:52.694016613 +0000 UTC m=+89.996320777" watchObservedRunningTime="2026-07-08 19:12:52.695614842 +0000 UTC m=+89.997919013" Jul 08 19:12:53.682022 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:53.681976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" event={"ID":"32905ee7-0c58-41b5-99ff-e42f59fc1039","Type":"ContainerStarted","Data":"87aa4adf0155be990fa62b24fe8bc7f2f6bd6b7b3afcaba53990b2c8a4bdb690"} Jul 08 19:12:53.698857 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:53.698769 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" podStartSLOduration=1.334891468 podStartE2EDuration="5.698756334s" totalImagesPullingTime="4.363864866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:53.697893953 +0000 UTC m=+91.000198130" watchObservedRunningTime="2026-07-08 19:12:53.698756334 +0000 UTC m=+91.001060501" Jul 08 19:12:54.685876 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:54.685840 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:12:54.687628 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:12:54.687606 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-784684cd49-29d9g" Jul 08 19:13:02.498112 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:02.498066 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:13:02.501214 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:02.501183 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"image-registry-5ff55887b9-vn4sp\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:13:02.586811 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:02.586782 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b27qk" Jul 08 19:13:02.715770 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:02.715742 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9x74r\"" Jul 08 19:13:02.723693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:02.723668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:13:02.894153 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:02.894027 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ff55887b9-vn4sp"] Jul 08 19:13:02.898700 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:13:02.898667 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cad1733_48a4_4a9b_bd22_fb2e7b35c008.slice/crio-9c680f14755dbf1867d82b025331d56488beefcf19ffcf88d3b5406c373d1682 WatchSource:0}: Error finding container 9c680f14755dbf1867d82b025331d56488beefcf19ffcf88d3b5406c373d1682: Status 404 returned error can't find the container with id 9c680f14755dbf1867d82b025331d56488beefcf19ffcf88d3b5406c373d1682 Jul 08 19:13:03.722010 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:03.721963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" event={"ID":"1cad1733-48a4-4a9b-bd22-fb2e7b35c008","Type":"ContainerStarted","Data":"4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05"} Jul 08 19:13:03.722529 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:03.722029 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" event={"ID":"1cad1733-48a4-4a9b-bd22-fb2e7b35c008","Type":"ContainerStarted","Data":"9c680f14755dbf1867d82b025331d56488beefcf19ffcf88d3b5406c373d1682"} Jul 08 19:13:03.722529 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:03.722075 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:13:03.749081 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:03.749027 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" podStartSLOduration=99.749008055 podStartE2EDuration="1m39.749008055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:03.746866398 +0000 UTC m=+101.049170566" watchObservedRunningTime="2026-07-08 19:13:03.749008055 +0000 UTC m=+101.051312225" Jul 08 19:13:07.807472 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.807442 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-29ws8"] Jul 08 19:13:07.841633 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.841597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:07.846280 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.846191 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 19:13:07.846457 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.846292 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 19:13:07.846457 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.846314 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 19:13:07.846457 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.846375 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 19:13:07.846457 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.846202 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 19:13:07.846457 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.846203 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 19:13:07.846457 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.846442 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pt55g\"" Jul 08 19:13:07.942330 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.942298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-textfile\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:07.942528 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.942363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7r6\" (UniqueName: \"kubernetes.io/projected/a1eb4547-ca65-447a-a218-3fc40397407c-kube-api-access-nw7r6\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:07.942528 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.942425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-wtmp\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:07.942528 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.942456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1eb4547-ca65-447a-a218-3fc40397407c-root\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:07.942528 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.942482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:07.942719 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.942558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1eb4547-ca65-447a-a218-3fc40397407c-sys\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:07.942719 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.942584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-tls\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:07.942719 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.942611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-accelerators-collector-config\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:07.942719 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:07.942662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb4547-ca65-447a-a218-3fc40397407c-metrics-client-ca\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043217 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb4547-ca65-447a-a218-3fc40397407c-metrics-client-ca\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043431 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-textfile\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043431 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7r6\" (UniqueName: \"kubernetes.io/projected/a1eb4547-ca65-447a-a218-3fc40397407c-kube-api-access-nw7r6\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043431 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043318 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-wtmp\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043431 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1eb4547-ca65-447a-a218-3fc40397407c-root\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043431 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043396 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1eb4547-ca65-447a-a218-3fc40397407c-sys\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-tls\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-accelerators-collector-config\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043595 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1eb4547-ca65-447a-a218-3fc40397407c-sys\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.043955 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043711 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1eb4547-ca65-447a-a218-3fc40397407c-root\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.044008 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.043995 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-textfile\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.044105 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.044078 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-wtmp\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.044157 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.044104 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb4547-ca65-447a-a218-3fc40397407c-metrics-client-ca\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.044157 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.044124 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-accelerators-collector-config\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.046278 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.046255 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.046422 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.046263 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1eb4547-ca65-447a-a218-3fc40397407c-node-exporter-tls\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.053568 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.053545 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7r6\" (UniqueName: \"kubernetes.io/projected/a1eb4547-ca65-447a-a218-3fc40397407c-kube-api-access-nw7r6\") pod \"node-exporter-29ws8\" (UID: \"a1eb4547-ca65-447a-a218-3fc40397407c\") " pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.152507 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.152417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-29ws8" Jul 08 19:13:08.191279 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:13:08.191234 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1eb4547_ca65_447a_a218_3fc40397407c.slice/crio-5ca4e854664888f0d8dfd5b73e90e3344969ca0bb3800a945a5240a86d4cf9b0 WatchSource:0}: Error finding container 5ca4e854664888f0d8dfd5b73e90e3344969ca0bb3800a945a5240a86d4cf9b0: Status 404 returned error can't find the container with id 5ca4e854664888f0d8dfd5b73e90e3344969ca0bb3800a945a5240a86d4cf9b0 Jul 08 19:13:08.738773 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:08.738741 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29ws8" event={"ID":"a1eb4547-ca65-447a-a218-3fc40397407c","Type":"ContainerStarted","Data":"5ca4e854664888f0d8dfd5b73e90e3344969ca0bb3800a945a5240a86d4cf9b0"} Jul 08 19:13:10.455254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:10.455221 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ff55887b9-vn4sp"] Jul 08 19:13:16.771480 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:16.771444 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1eb4547-ca65-447a-a218-3fc40397407c" containerID="cf4702513c9f096b6a3962b8c28f4135c5046329961f73a39bee91fe64f69b5d" exitCode=0 Jul 08 19:13:16.771798 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:16.771526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29ws8" event={"ID":"a1eb4547-ca65-447a-a218-3fc40397407c","Type":"ContainerDied","Data":"cf4702513c9f096b6a3962b8c28f4135c5046329961f73a39bee91fe64f69b5d"} Jul 08 19:13:17.778213 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:17.778086 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29ws8" event={"ID":"a1eb4547-ca65-447a-a218-3fc40397407c","Type":"ContainerStarted","Data":"1bc59850d2a6d8a432139eea98cc4bdc5dfbf463c7093aed8c05a27f0ca7e117"} Jul 08 19:13:17.778213 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:17.778165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-29ws8" event={"ID":"a1eb4547-ca65-447a-a218-3fc40397407c","Type":"ContainerStarted","Data":"e030bb4cbbfd0402e5bff207886fd1bf38f0e5860c1ab7cb0309e8ffd46f6a36"} Jul 08 19:13:17.779960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:17.779922 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-lljqn" event={"ID":"27d50fcc-7031-4354-8db6-023f93cad601","Type":"ContainerStarted","Data":"2e74b898778402fc82be642847ff8fa56d82f8513fe4e40c26b9248dfe1e5e6a"} Jul 08 19:13:17.780169 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:17.780143 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-lljqn" Jul 08 19:13:17.792961 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:17.792939 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-lljqn" Jul 08 19:13:17.802545 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:17.802493 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-29ws8" podStartSLOduration=9.269719642 podStartE2EDuration="10.802473943s" totalImagesPullingTime="1.467880082s" totalInitContainerRuntime="64.874219ms" isStatefulPod=true podCreationTimestamp="2026-07-08 19:13:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:17.800504965 +0000 UTC m=+115.102809144" watchObservedRunningTime="2026-07-08 19:13:17.802473943 +0000 UTC m=+115.104778111" Jul 08 19:13:17.819388 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:17.819325 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-lljqn" podStartSLOduration=2.302089662 podStartE2EDuration="29.819307173s" totalImagesPullingTime="27.517217511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:17.817847628 +0000 UTC m=+115.120151798" watchObservedRunningTime="2026-07-08 19:13:17.819307173 +0000 UTC m=+115.121611341" Jul 08 19:13:18.779525 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.779485 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5949c65448-t8nb8"] Jul 08 19:13:18.811176 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.811141 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5949c65448-t8nb8"] Jul 08 19:13:18.811366 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.811278 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:18.814729 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.814543 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hsc2j\"" Jul 08 19:13:18.814729 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.814570 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 19:13:18.814729 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.814600 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 19:13:18.814729 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.814667 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 19:13:18.816121 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.815760 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 19:13:18.816121 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.815760 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 19:13:18.820725 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.820705 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 19:13:18.940630 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.940594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56brx\" (UniqueName: \"kubernetes.io/projected/0a340780-aba7-4051-be28-6a520c4e14a4-kube-api-access-56brx\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:18.940800 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.940644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-service-ca\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:18.940800 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.940674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-trusted-ca-bundle\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:18.940800 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.940745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-serving-cert\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:18.940800 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.940765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-console-config\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:18.941015 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.940838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-oauth-config\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:18.941015 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:18.940893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-oauth-serving-cert\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.041638 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.041607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56brx\" (UniqueName: \"kubernetes.io/projected/0a340780-aba7-4051-be28-6a520c4e14a4-kube-api-access-56brx\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.041824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.041649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-service-ca\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.041824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.041667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-trusted-ca-bundle\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.041824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.041712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-serving-cert\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.041824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.041735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-console-config\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.041824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.041779 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-oauth-config\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.042093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.041824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-oauth-serving-cert\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.042768 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.042743 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-oauth-serving-cert\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.042943 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.042838 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-console-config\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.043068 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.042940 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-service-ca\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.043265 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.043242 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-trusted-ca-bundle\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.045183 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.045155 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-serving-cert\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.045282 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.045228 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-oauth-config\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.050519 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.050489 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56brx\" (UniqueName: \"kubernetes.io/projected/0a340780-aba7-4051-be28-6a520c4e14a4-kube-api-access-56brx\") pod \"console-5949c65448-t8nb8\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.123273 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.123234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:19.303570 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.303457 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5949c65448-t8nb8"] Jul 08 19:13:19.307419 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:13:19.307394 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a340780_aba7_4051_be28_6a520c4e14a4.slice/crio-1f41933065fbe69cdcef4bf010c23b265bf41c924addff3feeccdcf7ec38d550 WatchSource:0}: Error finding container 1f41933065fbe69cdcef4bf010c23b265bf41c924addff3feeccdcf7ec38d550: Status 404 returned error can't find the container with id 1f41933065fbe69cdcef4bf010c23b265bf41c924addff3feeccdcf7ec38d550 Jul 08 19:13:19.789405 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:19.789363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5949c65448-t8nb8" event={"ID":"0a340780-aba7-4051-be28-6a520c4e14a4","Type":"ContainerStarted","Data":"1f41933065fbe69cdcef4bf010c23b265bf41c924addff3feeccdcf7ec38d550"} Jul 08 19:13:20.463030 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:20.462996 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:13:24.822805 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:24.822765 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5949c65448-t8nb8" event={"ID":"0a340780-aba7-4051-be28-6a520c4e14a4","Type":"ContainerStarted","Data":"19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d"} Jul 08 19:13:24.842700 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:24.842639 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5949c65448-t8nb8" podStartSLOduration=2.403066215 podStartE2EDuration="6.842623665s" totalImagesPullingTime="4.43955745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:24.841079951 +0000 UTC m=+122.143384119" watchObservedRunningTime="2026-07-08 19:13:24.842623665 +0000 UTC m=+122.144927831" Jul 08 19:13:29.123425 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:29.123387 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:29.123869 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:29.123455 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:29.128449 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:29.128426 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:29.843759 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:29.843723 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:13:32.059939 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:32.059900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:13:32.062168 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:32.062145 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6-metrics-certs\") pod \"network-metrics-daemon-nfnnj\" (UID: \"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6\") " pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:13:32.277236 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:32.277207 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8qd8q\"" Jul 08 19:13:32.284666 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:32.284643 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nfnnj" Jul 08 19:13:32.455725 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:32.455703 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nfnnj"] Jul 08 19:13:32.459178 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:13:32.458894 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8f6c07_15b7_4e6c_9c8b_64f8dfc5cee6.slice/crio-5eb98c481a2e7ca2cfa96feeef4bb6d01f854e7dd40c8d744bbdf083c390d5e3 WatchSource:0}: Error finding container 5eb98c481a2e7ca2cfa96feeef4bb6d01f854e7dd40c8d744bbdf083c390d5e3: Status 404 returned error can't find the container with id 5eb98c481a2e7ca2cfa96feeef4bb6d01f854e7dd40c8d744bbdf083c390d5e3 Jul 08 19:13:32.850574 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:32.850542 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nfnnj" event={"ID":"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6","Type":"ContainerStarted","Data":"5eb98c481a2e7ca2cfa96feeef4bb6d01f854e7dd40c8d744bbdf083c390d5e3"} Jul 08 19:13:34.859400 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:34.859359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nfnnj" event={"ID":"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6","Type":"ContainerStarted","Data":"6a392558d41cd7d48095fa39e073becdb194a78d19788db18bbad452658e4870"} Jul 08 19:13:34.859799 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:34.859416 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nfnnj" event={"ID":"8b8f6c07-15b7-4e6c-9c8b-64f8dfc5cee6","Type":"ContainerStarted","Data":"b2284e7bc9e17b9df91e64910b3c799d65ad8d79ad523e1188f23b690d86d95d"} Jul 08 19:13:34.879465 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:34.879417 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nfnnj" podStartSLOduration=129.451343009 podStartE2EDuration="2m10.879402793s" totalImagesPullingTime="1.428059784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:34.877186197 +0000 UTC m=+132.179490363" watchObservedRunningTime="2026-07-08 19:13:34.879402793 +0000 UTC m=+132.181706959" Jul 08 19:13:35.477221 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.477156 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" podUID="1cad1733-48a4-4a9b-bd22-fb2e7b35c008" containerName="registry" containerID="cri-o://4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05" gracePeriod=30 Jul 08 19:13:35.772020 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.771991 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:13:35.864419 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.864386 2576 generic.go:356] "Generic (PLEG): container finished" podID="1cad1733-48a4-4a9b-bd22-fb2e7b35c008" containerID="4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05" exitCode=0 Jul 08 19:13:35.864840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.864433 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" Jul 08 19:13:35.864840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.864478 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" event={"ID":"1cad1733-48a4-4a9b-bd22-fb2e7b35c008","Type":"ContainerDied","Data":"4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05"} Jul 08 19:13:35.864840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.864533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ff55887b9-vn4sp" event={"ID":"1cad1733-48a4-4a9b-bd22-fb2e7b35c008","Type":"ContainerDied","Data":"9c680f14755dbf1867d82b025331d56488beefcf19ffcf88d3b5406c373d1682"} Jul 08 19:13:35.864840 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.864554 2576 scope.go:117] "RemoveContainer" containerID="4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05" Jul 08 19:13:35.872428 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.872408 2576 scope.go:117] "RemoveContainer" containerID="4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05" Jul 08 19:13:35.872691 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:13:35.872672 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05\": container with ID starting with 4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05 not found: ID does not exist" containerID="4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05" Jul 08 19:13:35.872777 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.872706 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05"} err="failed to get container status \"4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05\": rpc error: code = NotFound desc = could not find container \"4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05\": container with ID starting with 4da585c82ec800af1a13b79f8ee6d1b37eddaf362e01e78d89184447c80e7f05 not found: ID does not exist" Jul 08 19:13:35.889957 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.889935 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-trusted-ca\") pod \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " Jul 08 19:13:35.890046 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.889969 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-ca-trust-extracted\") pod \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " Jul 08 19:13:35.890046 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.889985 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-bound-sa-token\") pod \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " Jul 08 19:13:35.890046 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.890016 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-installation-pull-secrets\") pod \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " Jul 08 19:13:35.890178 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.890138 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-certificates\") pod \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " Jul 08 19:13:35.890230 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.890197 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") pod \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " Jul 08 19:13:35.890280 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.890236 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-image-registry-private-configuration\") pod \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " Jul 08 19:13:35.890334 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.890301 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r295j\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-kube-api-access-r295j\") pod \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\" (UID: \"1cad1733-48a4-4a9b-bd22-fb2e7b35c008\") " Jul 08 19:13:35.890752 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.890481 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1cad1733-48a4-4a9b-bd22-fb2e7b35c008" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:13:35.890752 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.890552 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1cad1733-48a4-4a9b-bd22-fb2e7b35c008" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:13:35.892626 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.892544 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1cad1733-48a4-4a9b-bd22-fb2e7b35c008" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:13:35.892626 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.892552 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1cad1733-48a4-4a9b-bd22-fb2e7b35c008" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:13:35.892794 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.892650 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1cad1733-48a4-4a9b-bd22-fb2e7b35c008" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:13:35.892938 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.892909 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1cad1733-48a4-4a9b-bd22-fb2e7b35c008" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:13:35.892998 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.892936 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-kube-api-access-r295j" (OuterVolumeSpecName: "kube-api-access-r295j") pod "1cad1733-48a4-4a9b-bd22-fb2e7b35c008" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008"). InnerVolumeSpecName "kube-api-access-r295j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:13:35.899255 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.899229 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1cad1733-48a4-4a9b-bd22-fb2e7b35c008" (UID: "1cad1733-48a4-4a9b-bd22-fb2e7b35c008"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:13:35.991401 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.991365 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-tls\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:13:35.991608 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.991455 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-image-registry-private-configuration\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:13:35.991608 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.991543 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r295j\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-kube-api-access-r295j\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:13:35.991608 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.991567 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-trusted-ca\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:13:35.991608 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.991578 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-ca-trust-extracted\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:13:35.991608 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.991587 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-bound-sa-token\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:13:35.991608 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.991595 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-installation-pull-secrets\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:13:35.991608 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:35.991604 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1cad1733-48a4-4a9b-bd22-fb2e7b35c008-registry-certificates\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:13:36.186286 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:36.186255 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ff55887b9-vn4sp"] Jul 08 19:13:36.191929 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:36.191902 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5ff55887b9-vn4sp"] Jul 08 19:13:37.266515 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:13:37.266486 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cad1733-48a4-4a9b-bd22-fb2e7b35c008" path="/var/lib/kubelet/pods/1cad1733-48a4-4a9b-bd22-fb2e7b35c008/volumes" Jul 08 19:14:35.666904 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.666869 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d495698fb-7dj6j"] Jul 08 19:14:35.667388 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.667118 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cad1733-48a4-4a9b-bd22-fb2e7b35c008" containerName="registry" Jul 08 19:14:35.667388 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.667128 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cad1733-48a4-4a9b-bd22-fb2e7b35c008" containerName="registry" Jul 08 19:14:35.667388 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.667180 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cad1733-48a4-4a9b-bd22-fb2e7b35c008" containerName="registry" Jul 08 19:14:35.669764 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.669745 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.687020 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.686988 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d495698fb-7dj6j"] Jul 08 19:14:35.717382 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.717339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-serving-cert\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.717529 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.717392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-service-ca\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.717529 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.717411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-config\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.717529 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.717520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-oauth-serving-cert\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.717700 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.717574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-trusted-ca-bundle\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.717700 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.717604 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhsdw\" (UniqueName: \"kubernetes.io/projected/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-kube-api-access-xhsdw\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.717700 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.717635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-oauth-config\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.818259 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.818219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-oauth-config\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.818259 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.818264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-serving-cert\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.818538 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.818287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-service-ca\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.818538 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.818411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-config\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.818538 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.818489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-oauth-serving-cert\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.818698 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.818541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-trusted-ca-bundle\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.818698 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.818566 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhsdw\" (UniqueName: \"kubernetes.io/projected/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-kube-api-access-xhsdw\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.819102 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.819079 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-service-ca\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.819210 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.819125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-config\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.819282 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.819221 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-oauth-serving-cert\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.819622 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.819591 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-trusted-ca-bundle\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.820831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.820798 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-oauth-config\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.820928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.820893 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-serving-cert\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.827573 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.827547 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhsdw\" (UniqueName: \"kubernetes.io/projected/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-kube-api-access-xhsdw\") pod \"console-7d495698fb-7dj6j\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:35.978695 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:35.978597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:36.100995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:36.100964 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d495698fb-7dj6j"] Jul 08 19:14:36.103711 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:14:36.103687 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d614a45_cc3e_4e9a_85e2_4cbf890ca7b1.slice/crio-2e1ebfe8f1960c32030c5b375675e3822dbfcbe69b63dd0424ed96c535c60e09 WatchSource:0}: Error finding container 2e1ebfe8f1960c32030c5b375675e3822dbfcbe69b63dd0424ed96c535c60e09: Status 404 returned error can't find the container with id 2e1ebfe8f1960c32030c5b375675e3822dbfcbe69b63dd0424ed96c535c60e09 Jul 08 19:14:37.084071 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:37.084035 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d495698fb-7dj6j" event={"ID":"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1","Type":"ContainerStarted","Data":"099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e"} Jul 08 19:14:37.084469 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:37.084087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d495698fb-7dj6j" event={"ID":"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1","Type":"ContainerStarted","Data":"2e1ebfe8f1960c32030c5b375675e3822dbfcbe69b63dd0424ed96c535c60e09"} Jul 08 19:14:37.106503 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:37.106452 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d495698fb-7dj6j" podStartSLOduration=2.106438468 podStartE2EDuration="2.106438468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:14:37.104851394 +0000 UTC m=+194.407155561" watchObservedRunningTime="2026-07-08 19:14:37.106438468 +0000 UTC m=+194.408742636" Jul 08 19:14:45.979094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:45.979057 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:45.979647 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:45.979121 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:45.984105 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:45.984082 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:46.119111 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:46.119081 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:14:46.168375 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:14:46.168323 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5949c65448-t8nb8"] Jul 08 19:15:11.196093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.196052 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5949c65448-t8nb8" podUID="0a340780-aba7-4051-be28-6a520c4e14a4" containerName="console" containerID="cri-o://19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d" gracePeriod=15 Jul 08 19:15:11.431932 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.431909 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:15:11.598910 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.598879 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-service-ca\") pod \"0a340780-aba7-4051-be28-6a520c4e14a4\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " Jul 08 19:15:11.599111 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.598926 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-oauth-serving-cert\") pod \"0a340780-aba7-4051-be28-6a520c4e14a4\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " Jul 08 19:15:11.599111 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.598950 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-serving-cert\") pod \"0a340780-aba7-4051-be28-6a520c4e14a4\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " Jul 08 19:15:11.599111 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.598986 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-trusted-ca-bundle\") pod \"0a340780-aba7-4051-be28-6a520c4e14a4\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " Jul 08 19:15:11.599111 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.599017 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56brx\" (UniqueName: \"kubernetes.io/projected/0a340780-aba7-4051-be28-6a520c4e14a4-kube-api-access-56brx\") pod \"0a340780-aba7-4051-be28-6a520c4e14a4\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " Jul 08 19:15:11.599111 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.599044 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-oauth-config\") pod \"0a340780-aba7-4051-be28-6a520c4e14a4\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " Jul 08 19:15:11.599111 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.599077 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-console-config\") pod \"0a340780-aba7-4051-be28-6a520c4e14a4\" (UID: \"0a340780-aba7-4051-be28-6a520c4e14a4\") " Jul 08 19:15:11.599519 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.599483 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0a340780-aba7-4051-be28-6a520c4e14a4" (UID: "0a340780-aba7-4051-be28-6a520c4e14a4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:15:11.599633 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.599546 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0a340780-aba7-4051-be28-6a520c4e14a4" (UID: "0a340780-aba7-4051-be28-6a520c4e14a4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:15:11.599633 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.599589 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-console-config" (OuterVolumeSpecName: "console-config") pod "0a340780-aba7-4051-be28-6a520c4e14a4" (UID: "0a340780-aba7-4051-be28-6a520c4e14a4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:15:11.599633 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.599619 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-service-ca" (OuterVolumeSpecName: "service-ca") pod "0a340780-aba7-4051-be28-6a520c4e14a4" (UID: "0a340780-aba7-4051-be28-6a520c4e14a4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:15:11.601451 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.601326 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0a340780-aba7-4051-be28-6a520c4e14a4" (UID: "0a340780-aba7-4051-be28-6a520c4e14a4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:15:11.601451 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.601396 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0a340780-aba7-4051-be28-6a520c4e14a4" (UID: "0a340780-aba7-4051-be28-6a520c4e14a4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:15:11.601451 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.601424 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a340780-aba7-4051-be28-6a520c4e14a4-kube-api-access-56brx" (OuterVolumeSpecName: "kube-api-access-56brx") pod "0a340780-aba7-4051-be28-6a520c4e14a4" (UID: "0a340780-aba7-4051-be28-6a520c4e14a4"). InnerVolumeSpecName "kube-api-access-56brx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:15:11.699564 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.699534 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56brx\" (UniqueName: \"kubernetes.io/projected/0a340780-aba7-4051-be28-6a520c4e14a4-kube-api-access-56brx\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:11.699564 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.699558 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-oauth-config\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:11.699564 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.699567 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-console-config\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:11.699766 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.699576 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-service-ca\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:11.699766 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.699584 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-oauth-serving-cert\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:11.699766 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.699592 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a340780-aba7-4051-be28-6a520c4e14a4-console-serving-cert\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:11.699766 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:11.699600 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a340780-aba7-4051-be28-6a520c4e14a4-trusted-ca-bundle\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:12.205672 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:12.205631 2576 generic.go:356] "Generic (PLEG): container finished" podID="0a340780-aba7-4051-be28-6a520c4e14a4" containerID="19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d" exitCode=0 Jul 08 19:15:12.205672 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:12.205672 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5949c65448-t8nb8" Jul 08 19:15:12.206140 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:12.205682 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5949c65448-t8nb8" event={"ID":"0a340780-aba7-4051-be28-6a520c4e14a4","Type":"ContainerDied","Data":"19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d"} Jul 08 19:15:12.206140 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:12.205729 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5949c65448-t8nb8" event={"ID":"0a340780-aba7-4051-be28-6a520c4e14a4","Type":"ContainerDied","Data":"1f41933065fbe69cdcef4bf010c23b265bf41c924addff3feeccdcf7ec38d550"} Jul 08 19:15:12.206140 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:12.205750 2576 scope.go:117] "RemoveContainer" containerID="19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d" Jul 08 19:15:12.214972 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:12.214948 2576 scope.go:117] "RemoveContainer" containerID="19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d" Jul 08 19:15:12.216512 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:15:12.216484 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d\": container with ID starting with 19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d not found: ID does not exist" containerID="19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d" Jul 08 19:15:12.216596 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:12.216524 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d"} err="failed to get container status \"19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d\": rpc error: code = NotFound desc = could not find container \"19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d\": container with ID starting with 19888b171440f1dd21b7e0c351dbea1a1fcaaad3a9b675a113636f7f1d69c70d not found: ID does not exist" Jul 08 19:15:12.229370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:12.229329 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5949c65448-t8nb8"] Jul 08 19:15:12.235231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:12.235211 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5949c65448-t8nb8"] Jul 08 19:15:13.266846 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:13.266809 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a340780-aba7-4051-be28-6a520c4e14a4" path="/var/lib/kubelet/pods/0a340780-aba7-4051-be28-6a520c4e14a4/volumes" Jul 08 19:15:16.316619 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.316581 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6"] Jul 08 19:15:16.317001 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.316835 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a340780-aba7-4051-be28-6a520c4e14a4" containerName="console" Jul 08 19:15:16.317001 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.316846 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a340780-aba7-4051-be28-6a520c4e14a4" containerName="console" Jul 08 19:15:16.317001 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.316890 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a340780-aba7-4051-be28-6a520c4e14a4" containerName="console" Jul 08 19:15:16.318900 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.318883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.321986 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.321963 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 19:15:16.322129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.321996 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fmsd4\"" Jul 08 19:15:16.322129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.322018 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 19:15:16.329850 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.329832 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6"] Jul 08 19:15:16.333969 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.333946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gkw7\" (UniqueName: \"kubernetes.io/projected/b5720ed9-0366-4d4c-b478-8049105a6101-kube-api-access-2gkw7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.334054 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.333995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.334094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.334061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.435094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.435057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gkw7\" (UniqueName: \"kubernetes.io/projected/b5720ed9-0366-4d4c-b478-8049105a6101-kube-api-access-2gkw7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.435094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.435098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.435313 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.435272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.435471 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.435444 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.435619 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.435601 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.446245 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.446217 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gkw7\" (UniqueName: \"kubernetes.io/projected/b5720ed9-0366-4d4c-b478-8049105a6101-kube-api-access-2gkw7\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.627641 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.627557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:16.756031 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:16.756009 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6"] Jul 08 19:15:16.758356 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:15:16.758316 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5720ed9_0366_4d4c_b478_8049105a6101.slice/crio-db41c574b9aad8434eede9d706f8a38a6a10d6c5f2237fb68a2c56d498118703 WatchSource:0}: Error finding container db41c574b9aad8434eede9d706f8a38a6a10d6c5f2237fb68a2c56d498118703: Status 404 returned error can't find the container with id db41c574b9aad8434eede9d706f8a38a6a10d6c5f2237fb68a2c56d498118703 Jul 08 19:15:17.223921 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:17.223888 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" event={"ID":"b5720ed9-0366-4d4c-b478-8049105a6101","Type":"ContainerStarted","Data":"db41c574b9aad8434eede9d706f8a38a6a10d6c5f2237fb68a2c56d498118703"} Jul 08 19:15:26.257907 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:26.257868 2576 generic.go:356] "Generic (PLEG): container finished" podID="b5720ed9-0366-4d4c-b478-8049105a6101" containerID="9efca2d2bf72fa6d3858880217f3fb8d7ee5cca158088eb1d790628cdc36aa60" exitCode=0 Jul 08 19:15:26.258280 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:26.257932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" event={"ID":"b5720ed9-0366-4d4c-b478-8049105a6101","Type":"ContainerDied","Data":"9efca2d2bf72fa6d3858880217f3fb8d7ee5cca158088eb1d790628cdc36aa60"} Jul 08 19:15:29.269194 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:29.269158 2576 generic.go:356] "Generic (PLEG): container finished" podID="b5720ed9-0366-4d4c-b478-8049105a6101" containerID="c885f12e43aa8baa5364bae490b3756248a0c211f1509c816f6e43a296a82df0" exitCode=0 Jul 08 19:15:29.269700 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:29.269245 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" event={"ID":"b5720ed9-0366-4d4c-b478-8049105a6101","Type":"ContainerDied","Data":"c885f12e43aa8baa5364bae490b3756248a0c211f1509c816f6e43a296a82df0"} Jul 08 19:15:36.296853 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:36.296816 2576 generic.go:356] "Generic (PLEG): container finished" podID="b5720ed9-0366-4d4c-b478-8049105a6101" containerID="576c42ab7c35ca8c7d9a9e0050ebe2ba98ae2e333620c0725482fd8826df192d" exitCode=0 Jul 08 19:15:36.297224 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:36.296901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" event={"ID":"b5720ed9-0366-4d4c-b478-8049105a6101","Type":"ContainerDied","Data":"576c42ab7c35ca8c7d9a9e0050ebe2ba98ae2e333620c0725482fd8826df192d"} Jul 08 19:15:37.419649 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:37.419623 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:37.504338 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:37.504306 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-bundle\") pod \"b5720ed9-0366-4d4c-b478-8049105a6101\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " Jul 08 19:15:37.504532 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:37.504436 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gkw7\" (UniqueName: \"kubernetes.io/projected/b5720ed9-0366-4d4c-b478-8049105a6101-kube-api-access-2gkw7\") pod \"b5720ed9-0366-4d4c-b478-8049105a6101\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " Jul 08 19:15:37.504532 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:37.504464 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-util\") pod \"b5720ed9-0366-4d4c-b478-8049105a6101\" (UID: \"b5720ed9-0366-4d4c-b478-8049105a6101\") " Jul 08 19:15:37.505174 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:37.505140 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-bundle" (OuterVolumeSpecName: "bundle") pod "b5720ed9-0366-4d4c-b478-8049105a6101" (UID: "b5720ed9-0366-4d4c-b478-8049105a6101"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:37.506704 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:37.506679 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5720ed9-0366-4d4c-b478-8049105a6101-kube-api-access-2gkw7" (OuterVolumeSpecName: "kube-api-access-2gkw7") pod "b5720ed9-0366-4d4c-b478-8049105a6101" (UID: "b5720ed9-0366-4d4c-b478-8049105a6101"). InnerVolumeSpecName "kube-api-access-2gkw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:15:37.509108 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:37.509084 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-util" (OuterVolumeSpecName: "util") pod "b5720ed9-0366-4d4c-b478-8049105a6101" (UID: "b5720ed9-0366-4d4c-b478-8049105a6101"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:37.605624 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:37.605539 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gkw7\" (UniqueName: \"kubernetes.io/projected/b5720ed9-0366-4d4c-b478-8049105a6101-kube-api-access-2gkw7\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:37.605624 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:37.605568 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-util\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:37.605624 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:37.605581 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5720ed9-0366-4d4c-b478-8049105a6101-bundle\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:38.305743 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:38.305719 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" Jul 08 19:15:38.305911 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:38.305719 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540br7t6" event={"ID":"b5720ed9-0366-4d4c-b478-8049105a6101","Type":"ContainerDied","Data":"db41c574b9aad8434eede9d706f8a38a6a10d6c5f2237fb68a2c56d498118703"} Jul 08 19:15:38.305911 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:38.305834 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db41c574b9aad8434eede9d706f8a38a6a10d6c5f2237fb68a2c56d498118703" Jul 08 19:15:49.151198 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.151159 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6"] Jul 08 19:15:49.151693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.151456 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5720ed9-0366-4d4c-b478-8049105a6101" containerName="extract" Jul 08 19:15:49.151693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.151467 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5720ed9-0366-4d4c-b478-8049105a6101" containerName="extract" Jul 08 19:15:49.151693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.151476 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5720ed9-0366-4d4c-b478-8049105a6101" containerName="pull" Jul 08 19:15:49.151693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.151482 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5720ed9-0366-4d4c-b478-8049105a6101" containerName="pull" Jul 08 19:15:49.151693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.151501 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5720ed9-0366-4d4c-b478-8049105a6101" containerName="util" Jul 08 19:15:49.151693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.151510 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5720ed9-0366-4d4c-b478-8049105a6101" containerName="util" Jul 08 19:15:49.151693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.151557 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5720ed9-0366-4d4c-b478-8049105a6101" containerName="extract" Jul 08 19:15:49.153531 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.153506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.156314 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.156295 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 19:15:49.156429 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.156378 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 19:15:49.157775 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.157755 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fmsd4\"" Jul 08 19:15:49.163547 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.163519 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6"] Jul 08 19:15:49.288451 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.288422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.288640 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.288493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b45hv\" (UniqueName: \"kubernetes.io/projected/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-kube-api-access-b45hv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.288640 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.288528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.389086 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.389051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45hv\" (UniqueName: \"kubernetes.io/projected/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-kube-api-access-b45hv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.389236 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.389091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.389274 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.389245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.389468 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.389453 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.389558 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.389542 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.397824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.397799 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45hv\" (UniqueName: \"kubernetes.io/projected/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-kube-api-access-b45hv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.462005 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.461932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:49.588857 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:49.588835 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6"] Jul 08 19:15:49.590848 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:15:49.590821 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a339c27_2dd1_4178_84ea_91ce4e8f59ae.slice/crio-15bbbe8ee15c8d75d2e13885ea84d85b3ddef6015ea3aef1051a5b8213ceb57f WatchSource:0}: Error finding container 15bbbe8ee15c8d75d2e13885ea84d85b3ddef6015ea3aef1051a5b8213ceb57f: Status 404 returned error can't find the container with id 15bbbe8ee15c8d75d2e13885ea84d85b3ddef6015ea3aef1051a5b8213ceb57f Jul 08 19:15:50.345857 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:50.345823 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a339c27-2dd1-4178-84ea-91ce4e8f59ae" containerID="a558f0cd55aeec43c3341e3764eb81fe5c8ab0d4d69f9edfbe5b4dd9116764f7" exitCode=0 Jul 08 19:15:50.346278 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:50.345903 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" event={"ID":"5a339c27-2dd1-4178-84ea-91ce4e8f59ae","Type":"ContainerDied","Data":"a558f0cd55aeec43c3341e3764eb81fe5c8ab0d4d69f9edfbe5b4dd9116764f7"} Jul 08 19:15:50.346278 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:50.345949 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" event={"ID":"5a339c27-2dd1-4178-84ea-91ce4e8f59ae","Type":"ContainerStarted","Data":"15bbbe8ee15c8d75d2e13885ea84d85b3ddef6015ea3aef1051a5b8213ceb57f"} Jul 08 19:15:53.361767 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:53.361725 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a339c27-2dd1-4178-84ea-91ce4e8f59ae" containerID="ebf1725b40d23bc70116dc845330b5512b993eefd042208072d53af2e6ee34b8" exitCode=0 Jul 08 19:15:53.362146 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:53.361804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" event={"ID":"5a339c27-2dd1-4178-84ea-91ce4e8f59ae","Type":"ContainerDied","Data":"ebf1725b40d23bc70116dc845330b5512b993eefd042208072d53af2e6ee34b8"} Jul 08 19:15:54.368020 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:54.367985 2576 generic.go:356] "Generic (PLEG): container finished" podID="5a339c27-2dd1-4178-84ea-91ce4e8f59ae" containerID="a1f2e4a96a29e749b78383c0da2d5d6e68e1d1bda354f43f3863487f315bebf8" exitCode=0 Jul 08 19:15:54.368425 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:54.368041 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" event={"ID":"5a339c27-2dd1-4178-84ea-91ce4e8f59ae","Type":"ContainerDied","Data":"a1f2e4a96a29e749b78383c0da2d5d6e68e1d1bda354f43f3863487f315bebf8"} Jul 08 19:15:55.486447 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:55.486415 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:55.532881 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:55.532847 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b45hv\" (UniqueName: \"kubernetes.io/projected/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-kube-api-access-b45hv\") pod \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " Jul 08 19:15:55.533023 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:55.532903 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-util\") pod \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " Jul 08 19:15:55.533023 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:55.532929 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-bundle\") pod \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\" (UID: \"5a339c27-2dd1-4178-84ea-91ce4e8f59ae\") " Jul 08 19:15:55.533334 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:55.533300 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-bundle" (OuterVolumeSpecName: "bundle") pod "5a339c27-2dd1-4178-84ea-91ce4e8f59ae" (UID: "5a339c27-2dd1-4178-84ea-91ce4e8f59ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:55.534888 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:55.534859 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-kube-api-access-b45hv" (OuterVolumeSpecName: "kube-api-access-b45hv") pod "5a339c27-2dd1-4178-84ea-91ce4e8f59ae" (UID: "5a339c27-2dd1-4178-84ea-91ce4e8f59ae"). InnerVolumeSpecName "kube-api-access-b45hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:15:55.537602 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:55.537585 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-util" (OuterVolumeSpecName: "util") pod "5a339c27-2dd1-4178-84ea-91ce4e8f59ae" (UID: "5a339c27-2dd1-4178-84ea-91ce4e8f59ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:55.634103 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:55.634032 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b45hv\" (UniqueName: \"kubernetes.io/projected/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-kube-api-access-b45hv\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:55.634103 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:55.634057 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-util\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:55.634103 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:55.634071 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5a339c27-2dd1-4178-84ea-91ce4e8f59ae-bundle\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:15:56.376153 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:56.376126 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" Jul 08 19:15:56.376153 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:56.376138 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbpdd6" event={"ID":"5a339c27-2dd1-4178-84ea-91ce4e8f59ae","Type":"ContainerDied","Data":"15bbbe8ee15c8d75d2e13885ea84d85b3ddef6015ea3aef1051a5b8213ceb57f"} Jul 08 19:15:56.376378 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:15:56.376184 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15bbbe8ee15c8d75d2e13885ea84d85b3ddef6015ea3aef1051a5b8213ceb57f" Jul 08 19:16:00.777778 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.777738 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-g2dz5"] Jul 08 19:16:00.778263 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.778127 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a339c27-2dd1-4178-84ea-91ce4e8f59ae" containerName="pull" Jul 08 19:16:00.778263 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.778145 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a339c27-2dd1-4178-84ea-91ce4e8f59ae" containerName="pull" Jul 08 19:16:00.778263 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.778160 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a339c27-2dd1-4178-84ea-91ce4e8f59ae" containerName="util" Jul 08 19:16:00.778263 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.778167 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a339c27-2dd1-4178-84ea-91ce4e8f59ae" containerName="util" Jul 08 19:16:00.778263 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.778193 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a339c27-2dd1-4178-84ea-91ce4e8f59ae" containerName="extract" Jul 08 19:16:00.778263 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.778201 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a339c27-2dd1-4178-84ea-91ce4e8f59ae" containerName="extract" Jul 08 19:16:00.778598 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.778271 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a339c27-2dd1-4178-84ea-91ce4e8f59ae" containerName="extract" Jul 08 19:16:00.780205 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.780184 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-g2dz5" Jul 08 19:16:00.782873 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.782847 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 19:16:00.782994 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.782976 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 19:16:00.784316 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.784300 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-78c4s\"" Jul 08 19:16:00.788620 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.788597 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-g2dz5"] Jul 08 19:16:00.875629 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.875597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4b2k\" (UniqueName: \"kubernetes.io/projected/13bb9e77-fca4-4679-aff7-80c0cd3d9056-kube-api-access-k4b2k\") pod \"cert-manager-55b4d954d5-g2dz5\" (UID: \"13bb9e77-fca4-4679-aff7-80c0cd3d9056\") " pod="cert-manager/cert-manager-55b4d954d5-g2dz5" Jul 08 19:16:00.875798 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.875642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13bb9e77-fca4-4679-aff7-80c0cd3d9056-bound-sa-token\") pod \"cert-manager-55b4d954d5-g2dz5\" (UID: \"13bb9e77-fca4-4679-aff7-80c0cd3d9056\") " pod="cert-manager/cert-manager-55b4d954d5-g2dz5" Jul 08 19:16:00.976360 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.976309 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4b2k\" (UniqueName: \"kubernetes.io/projected/13bb9e77-fca4-4679-aff7-80c0cd3d9056-kube-api-access-k4b2k\") pod \"cert-manager-55b4d954d5-g2dz5\" (UID: \"13bb9e77-fca4-4679-aff7-80c0cd3d9056\") " pod="cert-manager/cert-manager-55b4d954d5-g2dz5" Jul 08 19:16:00.976491 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.976366 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13bb9e77-fca4-4679-aff7-80c0cd3d9056-bound-sa-token\") pod \"cert-manager-55b4d954d5-g2dz5\" (UID: \"13bb9e77-fca4-4679-aff7-80c0cd3d9056\") " pod="cert-manager/cert-manager-55b4d954d5-g2dz5" Jul 08 19:16:00.985307 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.985278 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13bb9e77-fca4-4679-aff7-80c0cd3d9056-bound-sa-token\") pod \"cert-manager-55b4d954d5-g2dz5\" (UID: \"13bb9e77-fca4-4679-aff7-80c0cd3d9056\") " pod="cert-manager/cert-manager-55b4d954d5-g2dz5" Jul 08 19:16:00.985427 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:00.985340 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4b2k\" (UniqueName: \"kubernetes.io/projected/13bb9e77-fca4-4679-aff7-80c0cd3d9056-kube-api-access-k4b2k\") pod \"cert-manager-55b4d954d5-g2dz5\" (UID: \"13bb9e77-fca4-4679-aff7-80c0cd3d9056\") " pod="cert-manager/cert-manager-55b4d954d5-g2dz5" Jul 08 19:16:01.089269 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:01.089237 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-g2dz5" Jul 08 19:16:01.214534 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:01.214506 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-g2dz5"] Jul 08 19:16:01.216683 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:01.216654 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13bb9e77_fca4_4679_aff7_80c0cd3d9056.slice/crio-3a3ee3a8b1e4ad451388752078df236fe9ca04c7356babfebcd4b449a06be872 WatchSource:0}: Error finding container 3a3ee3a8b1e4ad451388752078df236fe9ca04c7356babfebcd4b449a06be872: Status 404 returned error can't find the container with id 3a3ee3a8b1e4ad451388752078df236fe9ca04c7356babfebcd4b449a06be872 Jul 08 19:16:01.395192 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:01.395103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-g2dz5" event={"ID":"13bb9e77-fca4-4679-aff7-80c0cd3d9056","Type":"ContainerStarted","Data":"3a3ee3a8b1e4ad451388752078df236fe9ca04c7356babfebcd4b449a06be872"} Jul 08 19:16:03.082644 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.082605 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s"] Jul 08 19:16:03.097537 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.097500 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s"] Jul 08 19:16:03.097717 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.097623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" Jul 08 19:16:03.100765 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.100739 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:16:03.101974 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.101947 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7ghdg\"" Jul 08 19:16:03.102094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.102010 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 19:16:03.193974 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.193942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6cd16c0-9ea7-4836-a978-3629e2a12005-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pmw9s\" (UID: \"c6cd16c0-9ea7-4836-a978-3629e2a12005\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" Jul 08 19:16:03.194153 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.193985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwv8k\" (UniqueName: \"kubernetes.io/projected/c6cd16c0-9ea7-4836-a978-3629e2a12005-kube-api-access-rwv8k\") pod \"openshift-lws-operator-fd8ccff4c-pmw9s\" (UID: \"c6cd16c0-9ea7-4836-a978-3629e2a12005\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" Jul 08 19:16:03.294657 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.294625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6cd16c0-9ea7-4836-a978-3629e2a12005-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pmw9s\" (UID: \"c6cd16c0-9ea7-4836-a978-3629e2a12005\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" Jul 08 19:16:03.294842 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.294670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwv8k\" (UniqueName: \"kubernetes.io/projected/c6cd16c0-9ea7-4836-a978-3629e2a12005-kube-api-access-rwv8k\") pod \"openshift-lws-operator-fd8ccff4c-pmw9s\" (UID: \"c6cd16c0-9ea7-4836-a978-3629e2a12005\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" Jul 08 19:16:03.295042 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.295017 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6cd16c0-9ea7-4836-a978-3629e2a12005-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pmw9s\" (UID: \"c6cd16c0-9ea7-4836-a978-3629e2a12005\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" Jul 08 19:16:03.303870 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.303848 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwv8k\" (UniqueName: \"kubernetes.io/projected/c6cd16c0-9ea7-4836-a978-3629e2a12005-kube-api-access-rwv8k\") pod \"openshift-lws-operator-fd8ccff4c-pmw9s\" (UID: \"c6cd16c0-9ea7-4836-a978-3629e2a12005\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" Jul 08 19:16:03.410314 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.410238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" Jul 08 19:16:03.948278 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:03.948202 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s"] Jul 08 19:16:03.950261 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:03.950232 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6cd16c0_9ea7_4836_a978_3629e2a12005.slice/crio-04260035f430cc3700c9a968b50afc6053db1bbddfb58c551f9390c7385dcdc9 WatchSource:0}: Error finding container 04260035f430cc3700c9a968b50afc6053db1bbddfb58c551f9390c7385dcdc9: Status 404 returned error can't find the container with id 04260035f430cc3700c9a968b50afc6053db1bbddfb58c551f9390c7385dcdc9 Jul 08 19:16:04.408310 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:04.408272 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-g2dz5" event={"ID":"13bb9e77-fca4-4679-aff7-80c0cd3d9056","Type":"ContainerStarted","Data":"64f8e3d9db09a8588767da37512f1255764952f82b030e22535809af2c7a7a06"} Jul 08 19:16:04.414303 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:04.414277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" event={"ID":"c6cd16c0-9ea7-4836-a978-3629e2a12005","Type":"ContainerStarted","Data":"04260035f430cc3700c9a968b50afc6053db1bbddfb58c551f9390c7385dcdc9"} Jul 08 19:16:04.428495 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:04.428453 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-g2dz5" podStartSLOduration=1.427372016 podStartE2EDuration="4.428439352s" totalImagesPullingTime="3.001067336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:04.426037077 +0000 UTC m=+281.728341258" watchObservedRunningTime="2026-07-08 19:16:04.428439352 +0000 UTC m=+281.730743518" Jul 08 19:16:06.423523 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:06.423468 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" event={"ID":"c6cd16c0-9ea7-4836-a978-3629e2a12005","Type":"ContainerStarted","Data":"030cbd4c7e3f98178571a71a6d739986427de4f8c8aa995945d3e605dcc0d6d3"} Jul 08 19:16:06.439833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:06.439653 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pmw9s" podStartSLOduration=1.170926588 podStartE2EDuration="3.439634291s" totalImagesPullingTime="2.268707703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:06.439074777 +0000 UTC m=+283.741378945" watchObservedRunningTime="2026-07-08 19:16:06.439634291 +0000 UTC m=+283.741938458" Jul 08 19:16:08.885038 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:08.885010 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf"] Jul 08 19:16:08.887285 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:08.887269 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:08.890014 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:08.889995 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 19:16:08.891366 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:08.891335 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 19:16:08.891436 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:08.891372 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fmsd4\"" Jul 08 19:16:08.897099 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:08.897075 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf"] Jul 08 19:16:09.038028 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.038000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:09.038216 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.038049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:09.038216 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.038073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxsjb\" (UniqueName: \"kubernetes.io/projected/00d6c3e1-1a81-457e-8443-ab03ab653b68-kube-api-access-lxsjb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:09.139601 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.139521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:09.139601 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.139575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:09.139601 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.139595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxsjb\" (UniqueName: \"kubernetes.io/projected/00d6c3e1-1a81-457e-8443-ab03ab653b68-kube-api-access-lxsjb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:09.139895 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.139875 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:09.139988 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.139968 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:09.148128 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.148103 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxsjb\" (UniqueName: \"kubernetes.io/projected/00d6c3e1-1a81-457e-8443-ab03ab653b68-kube-api-access-lxsjb\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:09.196011 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.195980 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:09.324288 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.324254 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf"] Jul 08 19:16:09.327887 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:09.327859 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d6c3e1_1a81_457e_8443_ab03ab653b68.slice/crio-dd45751ecd760d64be79bb1201b8a75ca329ebe083041fdd09a55c68d2986978 WatchSource:0}: Error finding container dd45751ecd760d64be79bb1201b8a75ca329ebe083041fdd09a55c68d2986978: Status 404 returned error can't find the container with id dd45751ecd760d64be79bb1201b8a75ca329ebe083041fdd09a55c68d2986978 Jul 08 19:16:09.436621 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.436595 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" event={"ID":"00d6c3e1-1a81-457e-8443-ab03ab653b68","Type":"ContainerStarted","Data":"0d62b92f5df8bb74f1d36ec9f18424f16f2b373583275504dd41ec04b69c1f35"} Jul 08 19:16:09.436721 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:09.436642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" event={"ID":"00d6c3e1-1a81-457e-8443-ab03ab653b68","Type":"ContainerStarted","Data":"dd45751ecd760d64be79bb1201b8a75ca329ebe083041fdd09a55c68d2986978"} Jul 08 19:16:10.442243 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:10.442152 2576 generic.go:356] "Generic (PLEG): container finished" podID="00d6c3e1-1a81-457e-8443-ab03ab653b68" containerID="0d62b92f5df8bb74f1d36ec9f18424f16f2b373583275504dd41ec04b69c1f35" exitCode=0 Jul 08 19:16:10.442243 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:10.442177 2576 generic.go:356] "Generic (PLEG): container finished" podID="00d6c3e1-1a81-457e-8443-ab03ab653b68" containerID="2875cfffdce248963aac312556a2c526a9dc0866ca8aa366c41b9fc35dfffa1c" exitCode=0 Jul 08 19:16:10.442662 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:10.442242 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" event={"ID":"00d6c3e1-1a81-457e-8443-ab03ab653b68","Type":"ContainerDied","Data":"0d62b92f5df8bb74f1d36ec9f18424f16f2b373583275504dd41ec04b69c1f35"} Jul 08 19:16:10.442662 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:10.442303 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" event={"ID":"00d6c3e1-1a81-457e-8443-ab03ab653b68","Type":"ContainerDied","Data":"2875cfffdce248963aac312556a2c526a9dc0866ca8aa366c41b9fc35dfffa1c"} Jul 08 19:16:11.447832 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:11.447797 2576 generic.go:356] "Generic (PLEG): container finished" podID="00d6c3e1-1a81-457e-8443-ab03ab653b68" containerID="e42079c3c3bbaa9b27e1ab8f899f91c48d7f8eaf6f9e5870d438058e6fe86b97" exitCode=0 Jul 08 19:16:11.448231 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:11.447887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" event={"ID":"00d6c3e1-1a81-457e-8443-ab03ab653b68","Type":"ContainerDied","Data":"e42079c3c3bbaa9b27e1ab8f899f91c48d7f8eaf6f9e5870d438058e6fe86b97"} Jul 08 19:16:12.575211 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:12.575184 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:12.665715 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:12.665689 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxsjb\" (UniqueName: \"kubernetes.io/projected/00d6c3e1-1a81-457e-8443-ab03ab653b68-kube-api-access-lxsjb\") pod \"00d6c3e1-1a81-457e-8443-ab03ab653b68\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " Jul 08 19:16:12.665854 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:12.665733 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-bundle\") pod \"00d6c3e1-1a81-457e-8443-ab03ab653b68\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " Jul 08 19:16:12.665854 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:12.665757 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-util\") pod \"00d6c3e1-1a81-457e-8443-ab03ab653b68\" (UID: \"00d6c3e1-1a81-457e-8443-ab03ab653b68\") " Jul 08 19:16:12.666422 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:12.666395 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-bundle" (OuterVolumeSpecName: "bundle") pod "00d6c3e1-1a81-457e-8443-ab03ab653b68" (UID: "00d6c3e1-1a81-457e-8443-ab03ab653b68"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:16:12.667700 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:12.667677 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d6c3e1-1a81-457e-8443-ab03ab653b68-kube-api-access-lxsjb" (OuterVolumeSpecName: "kube-api-access-lxsjb") pod "00d6c3e1-1a81-457e-8443-ab03ab653b68" (UID: "00d6c3e1-1a81-457e-8443-ab03ab653b68"). InnerVolumeSpecName "kube-api-access-lxsjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:16:12.670967 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:12.670942 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-util" (OuterVolumeSpecName: "util") pod "00d6c3e1-1a81-457e-8443-ab03ab653b68" (UID: "00d6c3e1-1a81-457e-8443-ab03ab653b68"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:16:12.766578 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:12.766494 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxsjb\" (UniqueName: \"kubernetes.io/projected/00d6c3e1-1a81-457e-8443-ab03ab653b68-kube-api-access-lxsjb\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:16:12.766578 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:12.766524 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-bundle\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:16:12.766578 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:12.766533 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00d6c3e1-1a81-457e-8443-ab03ab653b68-util\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:16:13.458178 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:13.458147 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" Jul 08 19:16:13.458178 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:13.458158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwvmzf" event={"ID":"00d6c3e1-1a81-457e-8443-ab03ab653b68","Type":"ContainerDied","Data":"dd45751ecd760d64be79bb1201b8a75ca329ebe083041fdd09a55c68d2986978"} Jul 08 19:16:13.458399 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:13.458206 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd45751ecd760d64be79bb1201b8a75ca329ebe083041fdd09a55c68d2986978" Jul 08 19:16:22.961198 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.961166 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz"] Jul 08 19:16:22.961604 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.961457 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d6c3e1-1a81-457e-8443-ab03ab653b68" containerName="extract" Jul 08 19:16:22.961604 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.961468 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d6c3e1-1a81-457e-8443-ab03ab653b68" containerName="extract" Jul 08 19:16:22.961604 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.961476 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d6c3e1-1a81-457e-8443-ab03ab653b68" containerName="pull" Jul 08 19:16:22.961604 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.961482 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d6c3e1-1a81-457e-8443-ab03ab653b68" containerName="pull" Jul 08 19:16:22.961604 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.961498 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d6c3e1-1a81-457e-8443-ab03ab653b68" containerName="util" Jul 08 19:16:22.961604 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.961505 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d6c3e1-1a81-457e-8443-ab03ab653b68" containerName="util" Jul 08 19:16:22.961604 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.961547 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d6c3e1-1a81-457e-8443-ab03ab653b68" containerName="extract" Jul 08 19:16:22.963577 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.963561 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:22.969002 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.968981 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 19:16:22.969336 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.969315 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 19:16:22.969486 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.969467 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 19:16:22.969627 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.969611 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9h9dk\"" Jul 08 19:16:22.969976 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:22.969824 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 19:16:23.003359 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.003301 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz"] Jul 08 19:16:23.098998 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.098967 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:16:23.100810 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.100790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:16:23.106602 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.106587 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 19:16:23.147832 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.147795 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skfkm\" (UniqueName: \"kubernetes.io/projected/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-kube-api-access-skfkm\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.147999 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.147846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-tmp\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.147999 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.147913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.147999 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.147993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-manifests\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.148113 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.148020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.249233 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.249146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.249233 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.249219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skfkm\" (UniqueName: \"kubernetes.io/projected/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-kube-api-access-skfkm\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.249497 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.249250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-tmp\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.249497 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.249291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.249497 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.249398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-manifests\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.249759 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.249735 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-manifests\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.251501 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.251479 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-tmp\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.252601 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.252587 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 19:16:23.259730 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.259711 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 19:16:23.261687 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.261666 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.261773 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.261694 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.270536 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.270516 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 19:16:23.280001 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.279984 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skfkm\" (UniqueName: \"kubernetes.io/projected/a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9-kube-api-access-skfkm\") pod \"opendatahub-operator-controller-manager-7dc4d9889-9ccgz\" (UID: \"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.576110 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.576080 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9h9dk\"" Jul 08 19:16:23.584018 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.584002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:23.711772 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:23.711730 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz"] Jul 08 19:16:23.714053 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:23.714015 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fe1b1d_1c1a_43f0_832b_8bc42e55d9d9.slice/crio-8d7bae2c4f96ef7e806207980df304a4e278ec2670b11b07c6fbec8c5114e23a WatchSource:0}: Error finding container 8d7bae2c4f96ef7e806207980df304a4e278ec2670b11b07c6fbec8c5114e23a: Status 404 returned error can't find the container with id 8d7bae2c4f96ef7e806207980df304a4e278ec2670b11b07c6fbec8c5114e23a Jul 08 19:16:24.503737 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:24.503691 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" event={"ID":"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9","Type":"ContainerStarted","Data":"8d7bae2c4f96ef7e806207980df304a4e278ec2670b11b07c6fbec8c5114e23a"} Jul 08 19:16:27.520303 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:27.520264 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" event={"ID":"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9","Type":"ContainerStarted","Data":"c4b0ce03a4bc25bc65d0b5caaae632cb90e86bb26439e99ae33365fded1c76ae"} Jul 08 19:16:28.526982 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:28.526944 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9" containerID="c4b0ce03a4bc25bc65d0b5caaae632cb90e86bb26439e99ae33365fded1c76ae" exitCode=0 Jul 08 19:16:28.527476 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:28.527009 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" event={"ID":"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9","Type":"ContainerDied","Data":"c4b0ce03a4bc25bc65d0b5caaae632cb90e86bb26439e99ae33365fded1c76ae"} Jul 08 19:16:28.527679 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:28.527657 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:16:29.532675 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:29.532639 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" event={"ID":"a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9","Type":"ContainerStarted","Data":"bf2ce3104bc8f7a0d240786d12b9c9685b2c509c3dbb777951a6374907528daf"} Jul 08 19:16:29.533050 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:29.532778 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:29.555951 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:29.555879 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" podStartSLOduration=3.096180391 podStartE2EDuration="7.555863596s" totalImagesPullingTime="3.39787679s" totalInitContainerRuntime="1.061806415s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:29.553718007 +0000 UTC m=+306.856022213" watchObservedRunningTime="2026-07-08 19:16:29.555863596 +0000 UTC m=+306.858167762" Jul 08 19:16:40.539495 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:40.539458 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dc4d9889-9ccgz" Jul 08 19:16:42.745985 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.745949 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw"] Jul 08 19:16:42.751631 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.751606 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:42.755830 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.755806 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 19:16:42.756036 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.756018 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-frxlr\"" Jul 08 19:16:42.758953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.758936 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 19:16:42.764323 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.764303 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw"] Jul 08 19:16:42.792092 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.792059 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e475a700-abbe-41f4-984b-5fcfbbb0ee27-tls-certs\") pod \"kube-auth-proxy-7d9fdb5547-lvgpw\" (UID: \"e475a700-abbe-41f4-984b-5fcfbbb0ee27\") " pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:42.792092 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.792093 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e475a700-abbe-41f4-984b-5fcfbbb0ee27-tmp\") pod \"kube-auth-proxy-7d9fdb5547-lvgpw\" (UID: \"e475a700-abbe-41f4-984b-5fcfbbb0ee27\") " pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:42.792267 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.792140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk54t\" (UniqueName: \"kubernetes.io/projected/e475a700-abbe-41f4-984b-5fcfbbb0ee27-kube-api-access-zk54t\") pod \"kube-auth-proxy-7d9fdb5547-lvgpw\" (UID: \"e475a700-abbe-41f4-984b-5fcfbbb0ee27\") " pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:42.893474 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.893434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e475a700-abbe-41f4-984b-5fcfbbb0ee27-tls-certs\") pod \"kube-auth-proxy-7d9fdb5547-lvgpw\" (UID: \"e475a700-abbe-41f4-984b-5fcfbbb0ee27\") " pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:42.893615 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.893479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e475a700-abbe-41f4-984b-5fcfbbb0ee27-tmp\") pod \"kube-auth-proxy-7d9fdb5547-lvgpw\" (UID: \"e475a700-abbe-41f4-984b-5fcfbbb0ee27\") " pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:42.893615 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.893549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk54t\" (UniqueName: \"kubernetes.io/projected/e475a700-abbe-41f4-984b-5fcfbbb0ee27-kube-api-access-zk54t\") pod \"kube-auth-proxy-7d9fdb5547-lvgpw\" (UID: \"e475a700-abbe-41f4-984b-5fcfbbb0ee27\") " pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:42.895780 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.895750 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e475a700-abbe-41f4-984b-5fcfbbb0ee27-tmp\") pod \"kube-auth-proxy-7d9fdb5547-lvgpw\" (UID: \"e475a700-abbe-41f4-984b-5fcfbbb0ee27\") " pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:42.896093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.896070 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e475a700-abbe-41f4-984b-5fcfbbb0ee27-tls-certs\") pod \"kube-auth-proxy-7d9fdb5547-lvgpw\" (UID: \"e475a700-abbe-41f4-984b-5fcfbbb0ee27\") " pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:42.909252 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:42.909229 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk54t\" (UniqueName: \"kubernetes.io/projected/e475a700-abbe-41f4-984b-5fcfbbb0ee27-kube-api-access-zk54t\") pod \"kube-auth-proxy-7d9fdb5547-lvgpw\" (UID: \"e475a700-abbe-41f4-984b-5fcfbbb0ee27\") " pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:43.040435 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.040398 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v7f6g"] Jul 08 19:16:43.042741 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.042718 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:16:43.045939 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.045916 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 08 19:16:43.046187 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.046173 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wxb86\"" Jul 08 19:16:43.061234 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.061201 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" Jul 08 19:16:43.065035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.065014 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v7f6g"] Jul 08 19:16:43.095966 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.095928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qszq\" (UniqueName: \"kubernetes.io/projected/5f0982f4-2a41-43fd-aa92-659927d24ad9-kube-api-access-2qszq\") pod \"odh-model-controller-7998c67dc4-v7f6g\" (UID: \"5f0982f4-2a41-43fd-aa92-659927d24ad9\") " pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:16:43.096113 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.095982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f0982f4-2a41-43fd-aa92-659927d24ad9-cert\") pod \"odh-model-controller-7998c67dc4-v7f6g\" (UID: \"5f0982f4-2a41-43fd-aa92-659927d24ad9\") " pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:16:43.196703 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.196665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qszq\" (UniqueName: \"kubernetes.io/projected/5f0982f4-2a41-43fd-aa92-659927d24ad9-kube-api-access-2qszq\") pod \"odh-model-controller-7998c67dc4-v7f6g\" (UID: \"5f0982f4-2a41-43fd-aa92-659927d24ad9\") " pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:16:43.196888 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.196724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f0982f4-2a41-43fd-aa92-659927d24ad9-cert\") pod \"odh-model-controller-7998c67dc4-v7f6g\" (UID: \"5f0982f4-2a41-43fd-aa92-659927d24ad9\") " pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:16:43.196888 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:16:43.196820 2576 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 08 19:16:43.197005 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:16:43.196890 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f0982f4-2a41-43fd-aa92-659927d24ad9-cert podName:5f0982f4-2a41-43fd-aa92-659927d24ad9 nodeName:}" failed. No retries permitted until 2026-07-08 19:16:43.696871265 +0000 UTC m=+320.999175410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5f0982f4-2a41-43fd-aa92-659927d24ad9-cert") pod "odh-model-controller-7998c67dc4-v7f6g" (UID: "5f0982f4-2a41-43fd-aa92-659927d24ad9") : secret "odh-model-controller-webhook-cert" not found Jul 08 19:16:43.197375 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.197333 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw"] Jul 08 19:16:43.199872 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:43.199847 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode475a700_abbe_41f4_984b_5fcfbbb0ee27.slice/crio-8025a9425a1e0fb17076e518353c954511e5e178cc5f8d28a7d5717d1180c505 WatchSource:0}: Error finding container 8025a9425a1e0fb17076e518353c954511e5e178cc5f8d28a7d5717d1180c505: Status 404 returned error can't find the container with id 8025a9425a1e0fb17076e518353c954511e5e178cc5f8d28a7d5717d1180c505 Jul 08 19:16:43.210208 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.210188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qszq\" (UniqueName: \"kubernetes.io/projected/5f0982f4-2a41-43fd-aa92-659927d24ad9-kube-api-access-2qszq\") pod \"odh-model-controller-7998c67dc4-v7f6g\" (UID: \"5f0982f4-2a41-43fd-aa92-659927d24ad9\") " pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:16:43.591521 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.591483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" event={"ID":"e475a700-abbe-41f4-984b-5fcfbbb0ee27","Type":"ContainerStarted","Data":"8025a9425a1e0fb17076e518353c954511e5e178cc5f8d28a7d5717d1180c505"} Jul 08 19:16:43.700965 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.700927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f0982f4-2a41-43fd-aa92-659927d24ad9-cert\") pod \"odh-model-controller-7998c67dc4-v7f6g\" (UID: \"5f0982f4-2a41-43fd-aa92-659927d24ad9\") " pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:16:43.703685 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.703663 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5f0982f4-2a41-43fd-aa92-659927d24ad9-cert\") pod \"odh-model-controller-7998c67dc4-v7f6g\" (UID: \"5f0982f4-2a41-43fd-aa92-659927d24ad9\") " pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:16:43.952855 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:43.952767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:16:44.096478 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.096449 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v7f6g"] Jul 08 19:16:44.099861 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:44.099829 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0982f4_2a41_43fd_aa92_659927d24ad9.slice/crio-4ea0be6919279ebce1c9cbc158c04d73429cce08841bc1c8b022c18554c2980a WatchSource:0}: Error finding container 4ea0be6919279ebce1c9cbc158c04d73429cce08841bc1c8b022c18554c2980a: Status 404 returned error can't find the container with id 4ea0be6919279ebce1c9cbc158c04d73429cce08841bc1c8b022c18554c2980a Jul 08 19:16:44.148146 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.148111 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xtlqw"] Jul 08 19:16:44.150682 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.150652 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:16:44.153538 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.153510 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 08 19:16:44.172965 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.172937 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xtlqw"] Jul 08 19:16:44.205466 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.205387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmj55\" (UniqueName: \"kubernetes.io/projected/b863103a-45b1-46b7-8ad3-1ae10ebd8033-kube-api-access-hmj55\") pod \"model-serving-api-54d89bd5fc-xtlqw\" (UID: \"b863103a-45b1-46b7-8ad3-1ae10ebd8033\") " pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:16:44.205466 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.205437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b863103a-45b1-46b7-8ad3-1ae10ebd8033-tls-certs\") pod \"model-serving-api-54d89bd5fc-xtlqw\" (UID: \"b863103a-45b1-46b7-8ad3-1ae10ebd8033\") " pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:16:44.306210 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.306172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmj55\" (UniqueName: \"kubernetes.io/projected/b863103a-45b1-46b7-8ad3-1ae10ebd8033-kube-api-access-hmj55\") pod \"model-serving-api-54d89bd5fc-xtlqw\" (UID: \"b863103a-45b1-46b7-8ad3-1ae10ebd8033\") " pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:16:44.306393 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.306229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b863103a-45b1-46b7-8ad3-1ae10ebd8033-tls-certs\") pod \"model-serving-api-54d89bd5fc-xtlqw\" (UID: \"b863103a-45b1-46b7-8ad3-1ae10ebd8033\") " pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:16:44.306393 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:16:44.306359 2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 08 19:16:44.306491 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:16:44.306437 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b863103a-45b1-46b7-8ad3-1ae10ebd8033-tls-certs podName:b863103a-45b1-46b7-8ad3-1ae10ebd8033 nodeName:}" failed. No retries permitted until 2026-07-08 19:16:44.806418099 +0000 UTC m=+322.108722269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b863103a-45b1-46b7-8ad3-1ae10ebd8033-tls-certs") pod "model-serving-api-54d89bd5fc-xtlqw" (UID: "b863103a-45b1-46b7-8ad3-1ae10ebd8033") : secret "model-serving-api-tls" not found Jul 08 19:16:44.323200 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.323172 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmj55\" (UniqueName: \"kubernetes.io/projected/b863103a-45b1-46b7-8ad3-1ae10ebd8033-kube-api-access-hmj55\") pod \"model-serving-api-54d89bd5fc-xtlqw\" (UID: \"b863103a-45b1-46b7-8ad3-1ae10ebd8033\") " pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:16:44.413236 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.413192 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb"] Jul 08 19:16:44.415496 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.415479 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.420518 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.420499 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 19:16:44.420634 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.420583 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 19:16:44.420794 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.420774 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 19:16:44.420880 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.420864 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-db88q\"" Jul 08 19:16:44.421051 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.421039 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 19:16:44.434831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.434806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb"] Jul 08 19:16:44.508238 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.508158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-cacerts\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.508238 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.508228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-token\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.508440 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.508261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.508440 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.508333 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxltz\" (UniqueName: \"kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-kube-api-access-wxltz\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.508440 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.508386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.508440 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.508419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-local-certs\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.508600 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.508464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.596912 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.596876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" event={"ID":"5f0982f4-2a41-43fd-aa92-659927d24ad9","Type":"ContainerStarted","Data":"4ea0be6919279ebce1c9cbc158c04d73429cce08841bc1c8b022c18554c2980a"} Jul 08 19:16:44.608968 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.608933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-cacerts\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.609103 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.608997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-token\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.609103 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.609020 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.609103 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.609075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxltz\" (UniqueName: \"kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-kube-api-access-wxltz\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.609267 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.609195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.609267 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.609232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-local-certs\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.609395 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.609369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.609709 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.609684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.611391 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.611367 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-cacerts\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.611575 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.611552 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.611575 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.611568 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-local-certs\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.611680 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.611568 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.618582 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.618564 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-token\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.618676 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.618587 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxltz\" (UniqueName: \"kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-kube-api-access-wxltz\") pod \"istiod-openshift-gateway-7574459656-r5hzb\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.726855 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.726814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:44.810824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.810795 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b863103a-45b1-46b7-8ad3-1ae10ebd8033-tls-certs\") pod \"model-serving-api-54d89bd5fc-xtlqw\" (UID: \"b863103a-45b1-46b7-8ad3-1ae10ebd8033\") " pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:16:44.813074 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.813046 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b863103a-45b1-46b7-8ad3-1ae10ebd8033-tls-certs\") pod \"model-serving-api-54d89bd5fc-xtlqw\" (UID: \"b863103a-45b1-46b7-8ad3-1ae10ebd8033\") " pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:16:44.853733 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:44.853524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb"] Jul 08 19:16:44.856059 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:44.856026 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54fb36a_8969_4926_ac3e_90d8cc3c0b76.slice/crio-e58a4c84f01dd4dba394492b10983b38eb0d6c92f873f09c2c67de06bff0ea41 WatchSource:0}: Error finding container e58a4c84f01dd4dba394492b10983b38eb0d6c92f873f09c2c67de06bff0ea41: Status 404 returned error can't find the container with id e58a4c84f01dd4dba394492b10983b38eb0d6c92f873f09c2c67de06bff0ea41 Jul 08 19:16:45.063509 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:45.063473 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:16:45.228223 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:45.227875 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-xtlqw"] Jul 08 19:16:45.232528 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:45.232474 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb863103a_45b1_46b7_8ad3_1ae10ebd8033.slice/crio-21304a7d2fdd8f695e7f8b36176fb1c45b732a92995eedc5d694e5defdd6ccc9 WatchSource:0}: Error finding container 21304a7d2fdd8f695e7f8b36176fb1c45b732a92995eedc5d694e5defdd6ccc9: Status 404 returned error can't find the container with id 21304a7d2fdd8f695e7f8b36176fb1c45b732a92995eedc5d694e5defdd6ccc9 Jul 08 19:16:45.606368 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:45.606263 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" event={"ID":"f54fb36a-8969-4926-ac3e-90d8cc3c0b76","Type":"ContainerStarted","Data":"e58a4c84f01dd4dba394492b10983b38eb0d6c92f873f09c2c67de06bff0ea41"} Jul 08 19:16:45.607754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:45.607689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" event={"ID":"b863103a-45b1-46b7-8ad3-1ae10ebd8033","Type":"ContainerStarted","Data":"21304a7d2fdd8f695e7f8b36176fb1c45b732a92995eedc5d694e5defdd6ccc9"} Jul 08 19:16:49.167847 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.167800 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:16:49.168175 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.167875 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:16:49.630509 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.630413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" event={"ID":"b863103a-45b1-46b7-8ad3-1ae10ebd8033","Type":"ContainerStarted","Data":"0d0ecd5b67b8def130020f4dde4a69d3878ccd9ca70d6c87e3bd1def8900ca8e"} Jul 08 19:16:49.630509 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.630484 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:16:49.632617 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.632577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" event={"ID":"5f0982f4-2a41-43fd-aa92-659927d24ad9","Type":"ContainerStarted","Data":"54721ea5dc5e5a9366daf5c0d6245b8642c196af5d87f929b32c4de898fcf47a"} Jul 08 19:16:49.632761 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.632714 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:16:49.634370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.634327 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" event={"ID":"e475a700-abbe-41f4-984b-5fcfbbb0ee27","Type":"ContainerStarted","Data":"e7a1ec177808b05e15cd5673e3197fdd763cbd34043d890cfd2e155986319984"} Jul 08 19:16:49.636251 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.636225 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" event={"ID":"f54fb36a-8969-4926-ac3e-90d8cc3c0b76","Type":"ContainerStarted","Data":"068933c3aa34e56e5571091e8baab3f7db6a1fe5ba6b1993face56a6558e49e9"} Jul 08 19:16:49.636380 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.636285 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:49.668720 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.668678 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" podStartSLOduration=1.728475266 podStartE2EDuration="5.668664292s" totalImagesPullingTime="3.940189026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:49.667195852 +0000 UTC m=+326.969500014" watchObservedRunningTime="2026-07-08 19:16:49.668664292 +0000 UTC m=+326.970968460" Jul 08 19:16:49.734255 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.734202 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" podStartSLOduration=1.425259899 podStartE2EDuration="5.734181122s" totalImagesPullingTime="4.308921223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:49.704371634 +0000 UTC m=+327.006675799" watchObservedRunningTime="2026-07-08 19:16:49.734181122 +0000 UTC m=+327.036485291" Jul 08 19:16:49.735773 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.735714 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" podStartSLOduration=1.6582628019999999 podStartE2EDuration="6.735699573s" totalImagesPullingTime="5.077436771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:49.733743574 +0000 UTC m=+327.036047742" watchObservedRunningTime="2026-07-08 19:16:49.735699573 +0000 UTC m=+327.038003739" Jul 08 19:16:49.764583 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:49.764531 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d9fdb5547-lvgpw" podStartSLOduration=1.8242262390000001 podStartE2EDuration="7.764515298s" totalImagesPullingTime="5.940289059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:49.762599926 +0000 UTC m=+327.064904115" watchObservedRunningTime="2026-07-08 19:16:49.764515298 +0000 UTC m=+327.066819466" Jul 08 19:16:50.602601 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.602560 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv"] Jul 08 19:16:50.604905 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.604883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.608401 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.608377 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4flgz\"" Jul 08 19:16:50.624191 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.624166 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv"] Jul 08 19:16:50.642844 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.642824 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:50.663183 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.663161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.663183 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.663190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.663375 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.663221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.663375 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.663355 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.663495 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.663427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.663556 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.663535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.663609 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.663597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.663865 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.663832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsfdb\" (UniqueName: \"kubernetes.io/projected/422aebe4-bd88-43f9-8193-e210f4c4b0b3-kube-api-access-rsfdb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.663963 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.663885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.764852 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.764810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765032 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.764877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765032 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.764911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765032 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsfdb\" (UniqueName: \"kubernetes.io/projected/422aebe4-bd88-43f9-8193-e210f4c4b0b3-kube-api-access-rsfdb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765147 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765287 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765333 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765619 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765581 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765700 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765628 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.765915 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.765891 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.767500 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.767477 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.767608 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.767589 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.776569 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.776548 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422aebe4-bd88-43f9-8193-e210f4c4b0b3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.776832 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.776811 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsfdb\" (UniqueName: \"kubernetes.io/projected/422aebe4-bd88-43f9-8193-e210f4c4b0b3-kube-api-access-rsfdb\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4cgrkv\" (UID: \"422aebe4-bd88-43f9-8193-e210f4c4b0b3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:50.915513 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:50.915418 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:51.107620 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:51.107597 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv"] Jul 08 19:16:51.109553 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:51.109527 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422aebe4_bd88_43f9_8193_e210f4c4b0b3.slice/crio-5500d6c77c3a719661fbc31d0c01fae59dc79177fd3aa87d4835573889ae4a94 WatchSource:0}: Error finding container 5500d6c77c3a719661fbc31d0c01fae59dc79177fd3aa87d4835573889ae4a94: Status 404 returned error can't find the container with id 5500d6c77c3a719661fbc31d0c01fae59dc79177fd3aa87d4835573889ae4a94 Jul 08 19:16:51.647550 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:51.647500 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" event={"ID":"422aebe4-bd88-43f9-8193-e210f4c4b0b3","Type":"ContainerStarted","Data":"5500d6c77c3a719661fbc31d0c01fae59dc79177fd3aa87d4835573889ae4a94"} Jul 08 19:16:54.168993 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.168956 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5"] Jul 08 19:16:54.171311 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.171296 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" Jul 08 19:16:54.171947 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.171916 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:16:54.172028 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.171978 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:16:54.172028 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.172010 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:16:54.177565 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.177546 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 08 19:16:54.177650 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.177546 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9bxjs\"" Jul 08 19:16:54.186012 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.185992 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq"] Jul 08 19:16:54.189160 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.189139 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" Jul 08 19:16:54.189564 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.189543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5"] Jul 08 19:16:54.191998 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.191975 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 08 19:16:54.192089 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.191983 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ml2f4\"" Jul 08 19:16:54.197625 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.197606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fca884ca-d632-498d-849d-b76609c5f1b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-6zsf5\" (UID: \"fca884ca-d632-498d-849d-b76609c5f1b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" Jul 08 19:16:54.197717 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.197651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlhd\" (UniqueName: \"kubernetes.io/projected/fca884ca-d632-498d-849d-b76609c5f1b8-kube-api-access-qmlhd\") pod \"kserve-controller-manager-d6f4b65b8-6zsf5\" (UID: \"fca884ca-d632-498d-849d-b76609c5f1b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" Jul 08 19:16:54.206480 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.206452 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq"] Jul 08 19:16:54.299086 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.299061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fca884ca-d632-498d-849d-b76609c5f1b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-6zsf5\" (UID: \"fca884ca-d632-498d-849d-b76609c5f1b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" Jul 08 19:16:54.299208 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.299145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlhd\" (UniqueName: \"kubernetes.io/projected/fca884ca-d632-498d-849d-b76609c5f1b8-kube-api-access-qmlhd\") pod \"kserve-controller-manager-d6f4b65b8-6zsf5\" (UID: \"fca884ca-d632-498d-849d-b76609c5f1b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" Jul 08 19:16:54.299208 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.299197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4rq\" (UniqueName: \"kubernetes.io/projected/2340417d-8d50-4bc8-b70a-5aa2f246432c-kube-api-access-zq4rq\") pod \"llmisvc-controller-manager-777c6bd7d5-2d2pq\" (UID: \"2340417d-8d50-4bc8-b70a-5aa2f246432c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" Jul 08 19:16:54.299402 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.299258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2340417d-8d50-4bc8-b70a-5aa2f246432c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2d2pq\" (UID: \"2340417d-8d50-4bc8-b70a-5aa2f246432c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" Jul 08 19:16:54.301400 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.301373 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fca884ca-d632-498d-849d-b76609c5f1b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-6zsf5\" (UID: \"fca884ca-d632-498d-849d-b76609c5f1b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" Jul 08 19:16:54.309831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.309802 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlhd\" (UniqueName: \"kubernetes.io/projected/fca884ca-d632-498d-849d-b76609c5f1b8-kube-api-access-qmlhd\") pod \"kserve-controller-manager-d6f4b65b8-6zsf5\" (UID: \"fca884ca-d632-498d-849d-b76609c5f1b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" Jul 08 19:16:54.400076 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.400043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4rq\" (UniqueName: \"kubernetes.io/projected/2340417d-8d50-4bc8-b70a-5aa2f246432c-kube-api-access-zq4rq\") pod \"llmisvc-controller-manager-777c6bd7d5-2d2pq\" (UID: \"2340417d-8d50-4bc8-b70a-5aa2f246432c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" Jul 08 19:16:54.400235 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.400101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2340417d-8d50-4bc8-b70a-5aa2f246432c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2d2pq\" (UID: \"2340417d-8d50-4bc8-b70a-5aa2f246432c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" Jul 08 19:16:54.402541 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.402520 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2340417d-8d50-4bc8-b70a-5aa2f246432c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-2d2pq\" (UID: \"2340417d-8d50-4bc8-b70a-5aa2f246432c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" Jul 08 19:16:54.410885 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.410858 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4rq\" (UniqueName: \"kubernetes.io/projected/2340417d-8d50-4bc8-b70a-5aa2f246432c-kube-api-access-zq4rq\") pod \"llmisvc-controller-manager-777c6bd7d5-2d2pq\" (UID: \"2340417d-8d50-4bc8-b70a-5aa2f246432c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" Jul 08 19:16:54.482916 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.482838 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" Jul 08 19:16:54.561588 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.561553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" Jul 08 19:16:54.639752 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.639729 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5"] Jul 08 19:16:54.641549 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:54.641521 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca884ca_d632_498d_849d_b76609c5f1b8.slice/crio-1470eb4cbb0137a89c264b6b62661a9eb1f94ec8e3456809144a486f34f974fe WatchSource:0}: Error finding container 1470eb4cbb0137a89c264b6b62661a9eb1f94ec8e3456809144a486f34f974fe: Status 404 returned error can't find the container with id 1470eb4cbb0137a89c264b6b62661a9eb1f94ec8e3456809144a486f34f974fe Jul 08 19:16:54.665619 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.665588 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" event={"ID":"fca884ca-d632-498d-849d-b76609c5f1b8","Type":"ContainerStarted","Data":"1470eb4cbb0137a89c264b6b62661a9eb1f94ec8e3456809144a486f34f974fe"} Jul 08 19:16:54.667438 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.667401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" event={"ID":"422aebe4-bd88-43f9-8193-e210f4c4b0b3","Type":"ContainerStarted","Data":"d2bb434412c8028a71ee4fe205646c8de94f56de6857790d9324dbf9dcfa8120"} Jul 08 19:16:54.754072 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.753963 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" podStartSLOduration=1.693927901 podStartE2EDuration="4.753946013s" totalImagesPullingTime="3.060018112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:54.749758631 +0000 UTC m=+332.052062798" watchObservedRunningTime="2026-07-08 19:16:54.753946013 +0000 UTC m=+332.056250242" Jul 08 19:16:54.774761 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.774738 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq"] Jul 08 19:16:54.776712 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:16:54.776686 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2340417d_8d50_4bc8_b70a_5aa2f246432c.slice/crio-7e09f4a4ac14413eee5ce6a9113556e8bc084104c7a368dff8fb8fd6a4f1d3a3 WatchSource:0}: Error finding container 7e09f4a4ac14413eee5ce6a9113556e8bc084104c7a368dff8fb8fd6a4f1d3a3: Status 404 returned error can't find the container with id 7e09f4a4ac14413eee5ce6a9113556e8bc084104c7a368dff8fb8fd6a4f1d3a3 Jul 08 19:16:54.916254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:54.916227 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:55.673805 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:55.673764 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" event={"ID":"2340417d-8d50-4bc8-b70a-5aa2f246432c","Type":"ContainerStarted","Data":"7e09f4a4ac14413eee5ce6a9113556e8bc084104c7a368dff8fb8fd6a4f1d3a3"} Jul 08 19:16:55.923320 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:55.923289 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:56.445753 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.445720 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb"] Jul 08 19:16:56.445966 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.445943 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" podUID="f54fb36a-8969-4926-ac3e-90d8cc3c0b76" containerName="discovery" containerID="cri-o://068933c3aa34e56e5571091e8baab3f7db6a1fe5ba6b1993face56a6558e49e9" gracePeriod=30 Jul 08 19:16:56.683978 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.683858 2576 generic.go:356] "Generic (PLEG): container finished" podID="f54fb36a-8969-4926-ac3e-90d8cc3c0b76" containerID="068933c3aa34e56e5571091e8baab3f7db6a1fe5ba6b1993face56a6558e49e9" exitCode=0 Jul 08 19:16:56.684575 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.684483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" event={"ID":"f54fb36a-8969-4926-ac3e-90d8cc3c0b76","Type":"ContainerDied","Data":"068933c3aa34e56e5571091e8baab3f7db6a1fe5ba6b1993face56a6558e49e9"} Jul 08 19:16:56.685038 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.684882 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:56.686323 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.686279 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4cgrkv" Jul 08 19:16:56.713867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.713801 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:56.822877 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.822841 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-cacerts\") pod \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " Jul 08 19:16:56.823039 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.822894 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-local-certs\") pod \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " Jul 08 19:16:56.823039 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.822945 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-kubeconfig\") pod \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " Jul 08 19:16:56.823039 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.822985 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-dns-cert\") pod \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " Jul 08 19:16:56.823039 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.823033 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxltz\" (UniqueName: \"kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-kube-api-access-wxltz\") pod \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " Jul 08 19:16:56.823255 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.823067 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-token\") pod \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " Jul 08 19:16:56.823255 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.823141 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-ca-configmap\") pod \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\" (UID: \"f54fb36a-8969-4926-ac3e-90d8cc3c0b76\") " Jul 08 19:16:56.823948 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.823904 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f54fb36a-8969-4926-ac3e-90d8cc3c0b76" (UID: "f54fb36a-8969-4926-ac3e-90d8cc3c0b76"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:16:56.826616 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.826589 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f54fb36a-8969-4926-ac3e-90d8cc3c0b76" (UID: "f54fb36a-8969-4926-ac3e-90d8cc3c0b76"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:16:56.827100 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.827042 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f54fb36a-8969-4926-ac3e-90d8cc3c0b76" (UID: "f54fb36a-8969-4926-ac3e-90d8cc3c0b76"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:16:56.827100 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.827065 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-cacerts" (OuterVolumeSpecName: "cacerts") pod "f54fb36a-8969-4926-ac3e-90d8cc3c0b76" (UID: "f54fb36a-8969-4926-ac3e-90d8cc3c0b76"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:16:56.827100 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.827053 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-local-certs" (OuterVolumeSpecName: "local-certs") pod "f54fb36a-8969-4926-ac3e-90d8cc3c0b76" (UID: "f54fb36a-8969-4926-ac3e-90d8cc3c0b76"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:16:56.827295 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.827257 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-kube-api-access-wxltz" (OuterVolumeSpecName: "kube-api-access-wxltz") pod "f54fb36a-8969-4926-ac3e-90d8cc3c0b76" (UID: "f54fb36a-8969-4926-ac3e-90d8cc3c0b76"). InnerVolumeSpecName "kube-api-access-wxltz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:16:56.827431 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.827403 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-token" (OuterVolumeSpecName: "istio-token") pod "f54fb36a-8969-4926-ac3e-90d8cc3c0b76" (UID: "f54fb36a-8969-4926-ac3e-90d8cc3c0b76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:16:56.924781 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.924734 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-token\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:16:56.924781 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.924769 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-ca-configmap\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:16:56.924781 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.924780 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-cacerts\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:16:56.924781 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.924789 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-local-certs\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:16:56.925095 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.924800 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-kubeconfig\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:16:56.925095 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.924807 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-istio-csr-dns-cert\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:16:56.925095 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:56.924817 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxltz\" (UniqueName: \"kubernetes.io/projected/f54fb36a-8969-4926-ac3e-90d8cc3c0b76-kube-api-access-wxltz\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:16:57.692204 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:57.692170 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" Jul 08 19:16:57.692204 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:57.692194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" event={"ID":"f54fb36a-8969-4926-ac3e-90d8cc3c0b76","Type":"ContainerDied","Data":"e58a4c84f01dd4dba394492b10983b38eb0d6c92f873f09c2c67de06bff0ea41"} Jul 08 19:16:57.692739 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:57.692265 2576 scope.go:117] "RemoveContainer" containerID="068933c3aa34e56e5571091e8baab3f7db6a1fe5ba6b1993face56a6558e49e9" Jul 08 19:16:57.718950 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:57.718918 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb"] Jul 08 19:16:57.723803 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:57.723778 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb"] Jul 08 19:16:58.698361 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:58.698309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" event={"ID":"2340417d-8d50-4bc8-b70a-5aa2f246432c","Type":"ContainerStarted","Data":"968cdfb800dc4044d60e7962f216d0abfcf9dea639fe3d93f8b1c6547c105dd5"} Jul 08 19:16:58.698777 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:58.698421 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" Jul 08 19:16:58.700701 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:58.700678 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" event={"ID":"fca884ca-d632-498d-849d-b76609c5f1b8","Type":"ContainerStarted","Data":"182901fe76aa71c731c21b9ce43b97979f1019770bd6156b761c1a46f09f2d00"} Jul 08 19:16:58.700843 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:58.700780 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" Jul 08 19:16:58.717707 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:58.717662 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" podStartSLOduration=0.926437186 podStartE2EDuration="4.717641663s" totalImagesPullingTime="3.791204477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:58.716505017 +0000 UTC m=+336.018809184" watchObservedRunningTime="2026-07-08 19:16:58.717641663 +0000 UTC m=+336.019945830" Jul 08 19:16:58.736256 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:58.736197 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" podStartSLOduration=0.80790304 podStartE2EDuration="4.736183283s" totalImagesPullingTime="3.928280243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:16:58.734323518 +0000 UTC m=+336.036627685" watchObservedRunningTime="2026-07-08 19:16:58.736183283 +0000 UTC m=+336.038487450" Jul 08 19:16:59.272283 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:16:59.272245 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f54fb36a-8969-4926-ac3e-90d8cc3c0b76" path="/var/lib/kubelet/pods/f54fb36a-8969-4926-ac3e-90d8cc3c0b76/volumes" Jul 08 19:17:00.643711 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:00.643682 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-v7f6g" Jul 08 19:17:00.646769 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:00.646748 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-xtlqw" Jul 08 19:17:01.642856 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:01.642819 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-r5hzb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.28:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 19:17:01.643042 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:01.642883 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-r5hzb" podUID="f54fb36a-8969-4926-ac3e-90d8cc3c0b76" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.28:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 19:17:29.707569 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:29.707537 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-2d2pq" Jul 08 19:17:29.710483 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:29.710457 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6zsf5" Jul 08 19:17:43.800825 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.800786 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tst9h"] Jul 08 19:17:43.801188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.801100 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54fb36a-8969-4926-ac3e-90d8cc3c0b76" containerName="discovery" Jul 08 19:17:43.801188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.801110 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54fb36a-8969-4926-ac3e-90d8cc3c0b76" containerName="discovery" Jul 08 19:17:43.801188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.801183 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54fb36a-8969-4926-ac3e-90d8cc3c0b76" containerName="discovery" Jul 08 19:17:43.814466 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.814427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tst9h"] Jul 08 19:17:43.814627 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.814563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tst9h" Jul 08 19:17:43.817541 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.817518 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 19:17:43.818705 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.818683 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 19:17:43.818826 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.818744 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-5vjph\"" Jul 08 19:17:43.818962 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.818936 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkjk5\" (UniqueName: \"kubernetes.io/projected/2d6456c9-6865-4db8-8ce8-60b28951da8f-kube-api-access-dkjk5\") pod \"kuadrant-operator-catalog-tst9h\" (UID: \"2d6456c9-6865-4db8-8ce8-60b28951da8f\") " pod="kuadrant-system/kuadrant-operator-catalog-tst9h" Jul 08 19:17:43.919993 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.919959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjk5\" (UniqueName: \"kubernetes.io/projected/2d6456c9-6865-4db8-8ce8-60b28951da8f-kube-api-access-dkjk5\") pod \"kuadrant-operator-catalog-tst9h\" (UID: \"2d6456c9-6865-4db8-8ce8-60b28951da8f\") " pod="kuadrant-system/kuadrant-operator-catalog-tst9h" Jul 08 19:17:43.928522 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:43.928498 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjk5\" (UniqueName: \"kubernetes.io/projected/2d6456c9-6865-4db8-8ce8-60b28951da8f-kube-api-access-dkjk5\") pod \"kuadrant-operator-catalog-tst9h\" (UID: \"2d6456c9-6865-4db8-8ce8-60b28951da8f\") " pod="kuadrant-system/kuadrant-operator-catalog-tst9h" Jul 08 19:17:44.124699 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.124592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tst9h" Jul 08 19:17:44.166978 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.166945 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tst9h"] Jul 08 19:17:44.252195 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.252168 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tst9h"] Jul 08 19:17:44.254549 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:17:44.254517 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6456c9_6865_4db8_8ce8_60b28951da8f.slice/crio-f1f81e145d17b1a5f836dfefa1359b4816c80ea9ea070604365c8ac7562d8893 WatchSource:0}: Error finding container f1f81e145d17b1a5f836dfefa1359b4816c80ea9ea070604365c8ac7562d8893: Status 404 returned error can't find the container with id f1f81e145d17b1a5f836dfefa1359b4816c80ea9ea070604365c8ac7562d8893 Jul 08 19:17:44.378380 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.378290 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v8b52"] Jul 08 19:17:44.381584 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.381567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v8b52" Jul 08 19:17:44.389289 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.389265 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v8b52"] Jul 08 19:17:44.424112 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.424078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqj7g\" (UniqueName: \"kubernetes.io/projected/5cfd158d-de8b-4588-a011-a67ce68db14e-kube-api-access-cqj7g\") pod \"kuadrant-operator-catalog-v8b52\" (UID: \"5cfd158d-de8b-4588-a011-a67ce68db14e\") " pod="kuadrant-system/kuadrant-operator-catalog-v8b52" Jul 08 19:17:44.524583 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.524526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqj7g\" (UniqueName: \"kubernetes.io/projected/5cfd158d-de8b-4588-a011-a67ce68db14e-kube-api-access-cqj7g\") pod \"kuadrant-operator-catalog-v8b52\" (UID: \"5cfd158d-de8b-4588-a011-a67ce68db14e\") " pod="kuadrant-system/kuadrant-operator-catalog-v8b52" Jul 08 19:17:44.533288 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.533259 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqj7g\" (UniqueName: \"kubernetes.io/projected/5cfd158d-de8b-4588-a011-a67ce68db14e-kube-api-access-cqj7g\") pod \"kuadrant-operator-catalog-v8b52\" (UID: \"5cfd158d-de8b-4588-a011-a67ce68db14e\") " pod="kuadrant-system/kuadrant-operator-catalog-v8b52" Jul 08 19:17:44.691725 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.691650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-v8b52" Jul 08 19:17:44.822957 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.822912 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-v8b52"] Jul 08 19:17:44.855025 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:17:44.854995 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfd158d_de8b_4588_a011_a67ce68db14e.slice/crio-ff032c0fa78c2dc06bed3b62a65bf9aff2e14b8d1573f0593e735e8891466986 WatchSource:0}: Error finding container ff032c0fa78c2dc06bed3b62a65bf9aff2e14b8d1573f0593e735e8891466986: Status 404 returned error can't find the container with id ff032c0fa78c2dc06bed3b62a65bf9aff2e14b8d1573f0593e735e8891466986 Jul 08 19:17:44.911297 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.911255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tst9h" event={"ID":"2d6456c9-6865-4db8-8ce8-60b28951da8f","Type":"ContainerStarted","Data":"f1f81e145d17b1a5f836dfefa1359b4816c80ea9ea070604365c8ac7562d8893"} Jul 08 19:17:44.912524 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:44.912489 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v8b52" event={"ID":"5cfd158d-de8b-4588-a011-a67ce68db14e","Type":"ContainerStarted","Data":"ff032c0fa78c2dc06bed3b62a65bf9aff2e14b8d1573f0593e735e8891466986"} Jul 08 19:17:46.925801 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:46.925761 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tst9h" event={"ID":"2d6456c9-6865-4db8-8ce8-60b28951da8f","Type":"ContainerStarted","Data":"28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391"} Jul 08 19:17:46.926293 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:46.925793 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-tst9h" podUID="2d6456c9-6865-4db8-8ce8-60b28951da8f" containerName="registry-server" containerID="cri-o://28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391" gracePeriod=2 Jul 08 19:17:46.927410 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:46.927380 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-v8b52" event={"ID":"5cfd158d-de8b-4588-a011-a67ce68db14e","Type":"ContainerStarted","Data":"12fcd7138689fa97dfd011c355ef4f005bffdcbb4b963953d864cf128fa7f84a"} Jul 08 19:17:46.945733 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:46.945690 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tst9h" podStartSLOduration=1.8434702459999999 podStartE2EDuration="3.945674543s" totalImagesPullingTime="2.102204297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:17:46.942528833 +0000 UTC m=+384.244833000" watchObservedRunningTime="2026-07-08 19:17:46.945674543 +0000 UTC m=+384.247978730" Jul 08 19:17:46.960180 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:46.960137 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-v8b52" podStartSLOduration=1.461940728 podStartE2EDuration="2.960124058s" totalImagesPullingTime="1.49818333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:17:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:17:46.958927899 +0000 UTC m=+384.261232066" watchObservedRunningTime="2026-07-08 19:17:46.960124058 +0000 UTC m=+384.262428270" Jul 08 19:17:47.162269 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.162245 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tst9h" Jul 08 19:17:47.248382 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.248277 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkjk5\" (UniqueName: \"kubernetes.io/projected/2d6456c9-6865-4db8-8ce8-60b28951da8f-kube-api-access-dkjk5\") pod \"2d6456c9-6865-4db8-8ce8-60b28951da8f\" (UID: \"2d6456c9-6865-4db8-8ce8-60b28951da8f\") " Jul 08 19:17:47.250465 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.250433 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6456c9-6865-4db8-8ce8-60b28951da8f-kube-api-access-dkjk5" (OuterVolumeSpecName: "kube-api-access-dkjk5") pod "2d6456c9-6865-4db8-8ce8-60b28951da8f" (UID: "2d6456c9-6865-4db8-8ce8-60b28951da8f"). InnerVolumeSpecName "kube-api-access-dkjk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:17:47.349338 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.349302 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkjk5\" (UniqueName: \"kubernetes.io/projected/2d6456c9-6865-4db8-8ce8-60b28951da8f-kube-api-access-dkjk5\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:17:47.932586 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.932554 2576 generic.go:356] "Generic (PLEG): container finished" podID="2d6456c9-6865-4db8-8ce8-60b28951da8f" containerID="28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391" exitCode=0 Jul 08 19:17:47.932990 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.932609 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tst9h" Jul 08 19:17:47.932990 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.932639 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tst9h" event={"ID":"2d6456c9-6865-4db8-8ce8-60b28951da8f","Type":"ContainerDied","Data":"28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391"} Jul 08 19:17:47.932990 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.932694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tst9h" event={"ID":"2d6456c9-6865-4db8-8ce8-60b28951da8f","Type":"ContainerDied","Data":"f1f81e145d17b1a5f836dfefa1359b4816c80ea9ea070604365c8ac7562d8893"} Jul 08 19:17:47.932990 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.932715 2576 scope.go:117] "RemoveContainer" containerID="28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391" Jul 08 19:17:47.941262 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.941241 2576 scope.go:117] "RemoveContainer" containerID="28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391" Jul 08 19:17:47.941521 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:17:47.941505 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391\": container with ID starting with 28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391 not found: ID does not exist" containerID="28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391" Jul 08 19:17:47.941571 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.941528 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391"} err="failed to get container status \"28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391\": rpc error: code = NotFound desc = could not find container \"28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391\": container with ID starting with 28cf76f21814bb4be2d0f3f8b71068528ea4d63e727655a52b7653651edb1391 not found: ID does not exist" Jul 08 19:17:47.952504 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.952479 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tst9h"] Jul 08 19:17:47.956669 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:47.956632 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tst9h"] Jul 08 19:17:49.267581 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:49.267540 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d6456c9-6865-4db8-8ce8-60b28951da8f" path="/var/lib/kubelet/pods/2d6456c9-6865-4db8-8ce8-60b28951da8f/volumes" Jul 08 19:17:54.692338 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:54.692306 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-v8b52" Jul 08 19:17:54.692758 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:54.692380 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-v8b52" Jul 08 19:17:54.714849 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:54.714824 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-v8b52" Jul 08 19:17:54.987435 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:54.987363 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-v8b52" Jul 08 19:17:59.223195 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.223154 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd"] Jul 08 19:17:59.223725 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.223706 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d6456c9-6865-4db8-8ce8-60b28951da8f" containerName="registry-server" Jul 08 19:17:59.223805 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.223728 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d6456c9-6865-4db8-8ce8-60b28951da8f" containerName="registry-server" Jul 08 19:17:59.223860 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.223825 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d6456c9-6865-4db8-8ce8-60b28951da8f" containerName="registry-server" Jul 08 19:17:59.227175 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.227152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.229953 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.229935 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kk9wf\"" Jul 08 19:17:59.236800 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.236776 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd"] Jul 08 19:17:59.342486 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.342456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.342637 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.342494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wg47\" (UniqueName: \"kubernetes.io/projected/40918d12-8097-4ba3-b1eb-d13ec143f851-kube-api-access-9wg47\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.342637 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.342513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.443536 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.443499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.443735 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.443550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wg47\" (UniqueName: \"kubernetes.io/projected/40918d12-8097-4ba3-b1eb-d13ec143f851-kube-api-access-9wg47\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.443735 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.443591 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.443920 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.443896 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.443987 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.443946 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.451894 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.451866 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wg47\" (UniqueName: \"kubernetes.io/projected/40918d12-8097-4ba3-b1eb-d13ec143f851-kube-api-access-9wg47\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.536128 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.536100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:17:59.616306 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.615514 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds"] Jul 08 19:17:59.618369 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.618332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.627241 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.627214 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds"] Jul 08 19:17:59.644923 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.644900 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.645133 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.645113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xshw4\" (UniqueName: \"kubernetes.io/projected/3317a351-ff9f-4255-a3c4-282552c4c483-kube-api-access-xshw4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.645227 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.645175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.665382 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.665360 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd"] Jul 08 19:17:59.668313 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:17:59.668290 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40918d12_8097_4ba3_b1eb_d13ec143f851.slice/crio-219af02224b61460f68fdfbd74a68df00802ec233ec3026af11538d08d71eda2 WatchSource:0}: Error finding container 219af02224b61460f68fdfbd74a68df00802ec233ec3026af11538d08d71eda2: Status 404 returned error can't find the container with id 219af02224b61460f68fdfbd74a68df00802ec233ec3026af11538d08d71eda2 Jul 08 19:17:59.746556 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.746519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xshw4\" (UniqueName: \"kubernetes.io/projected/3317a351-ff9f-4255-a3c4-282552c4c483-kube-api-access-xshw4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.746729 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.746712 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.746805 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.746787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.747135 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.747114 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.747176 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.747146 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.755288 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.755258 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xshw4\" (UniqueName: \"kubernetes.io/projected/3317a351-ff9f-4255-a3c4-282552c4c483-kube-api-access-xshw4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.930883 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.930798 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:17:59.990836 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.990805 2576 generic.go:356] "Generic (PLEG): container finished" podID="40918d12-8097-4ba3-b1eb-d13ec143f851" containerID="c3d943db8efc499415ab121a4cd7f5b5eae3730418b540edb4578ee896de247f" exitCode=0 Jul 08 19:17:59.990960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.990866 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" event={"ID":"40918d12-8097-4ba3-b1eb-d13ec143f851","Type":"ContainerDied","Data":"c3d943db8efc499415ab121a4cd7f5b5eae3730418b540edb4578ee896de247f"} Jul 08 19:17:59.990960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:17:59.990912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" event={"ID":"40918d12-8097-4ba3-b1eb-d13ec143f851","Type":"ContainerStarted","Data":"219af02224b61460f68fdfbd74a68df00802ec233ec3026af11538d08d71eda2"} Jul 08 19:18:00.022072 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.022038 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd"] Jul 08 19:18:00.026141 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.026121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.037169 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.037146 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd"] Jul 08 19:18:00.049968 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.049942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.050100 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.050080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frprq\" (UniqueName: \"kubernetes.io/projected/e8c49737-2a60-45a5-a337-9cd86ae76e58-kube-api-access-frprq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.050145 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.050118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.067668 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.067647 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds"] Jul 08 19:18:00.069256 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:18:00.069235 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3317a351_ff9f_4255_a3c4_282552c4c483.slice/crio-acc79dbf4fe35a98dc2573652d68203350823f0f34c00adf31b1fd7adeaed188 WatchSource:0}: Error finding container acc79dbf4fe35a98dc2573652d68203350823f0f34c00adf31b1fd7adeaed188: Status 404 returned error can't find the container with id acc79dbf4fe35a98dc2573652d68203350823f0f34c00adf31b1fd7adeaed188 Jul 08 19:18:00.150973 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.150936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frprq\" (UniqueName: \"kubernetes.io/projected/e8c49737-2a60-45a5-a337-9cd86ae76e58-kube-api-access-frprq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.150973 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.150973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.151170 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.151018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.151477 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.151457 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.151513 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.151476 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.160790 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.160770 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frprq\" (UniqueName: \"kubernetes.io/projected/e8c49737-2a60-45a5-a337-9cd86ae76e58-kube-api-access-frprq\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.336410 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.336377 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:00.424113 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.424070 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8"] Jul 08 19:18:00.426842 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.426825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.435691 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.435524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8"] Jul 08 19:18:00.453676 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.453640 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7md4\" (UniqueName: \"kubernetes.io/projected/5538bfe3-3461-414b-bc7e-dfef1e582edb-kube-api-access-v7md4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.453830 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.453721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.453830 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.453807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.476941 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.473571 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd"] Jul 08 19:18:00.479837 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:18:00.479813 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c49737_2a60_45a5_a337_9cd86ae76e58.slice/crio-e91c5c8a0b087e809f859036d1bc4fe626fe694c7c1f477c3559a5a075fc1b06 WatchSource:0}: Error finding container e91c5c8a0b087e809f859036d1bc4fe626fe694c7c1f477c3559a5a075fc1b06: Status 404 returned error can't find the container with id e91c5c8a0b087e809f859036d1bc4fe626fe694c7c1f477c3559a5a075fc1b06 Jul 08 19:18:00.554412 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.554379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.554538 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.554446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.554538 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.554495 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7md4\" (UniqueName: \"kubernetes.io/projected/5538bfe3-3461-414b-bc7e-dfef1e582edb-kube-api-access-v7md4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.554843 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.554821 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.554914 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.554820 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.563954 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.563927 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7md4\" (UniqueName: \"kubernetes.io/projected/5538bfe3-3461-414b-bc7e-dfef1e582edb-kube-api-access-v7md4\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.739506 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.739436 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:00.878686 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.878665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8"] Jul 08 19:18:00.894158 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:18:00.894129 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5538bfe3_3461_414b_bc7e_dfef1e582edb.slice/crio-bbf9a1aedc7cefd63d1b6d62a8924575f7eefb65080f47f57b55499159bed28f WatchSource:0}: Error finding container bbf9a1aedc7cefd63d1b6d62a8924575f7eefb65080f47f57b55499159bed28f: Status 404 returned error can't find the container with id bbf9a1aedc7cefd63d1b6d62a8924575f7eefb65080f47f57b55499159bed28f Jul 08 19:18:00.996741 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.996675 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8c49737-2a60-45a5-a337-9cd86ae76e58" containerID="7732ddd57e4ba412c592d28e5c00601895b5f9ab906f82b8412755bb317dcf19" exitCode=0 Jul 08 19:18:00.996969 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.996753 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" event={"ID":"e8c49737-2a60-45a5-a337-9cd86ae76e58","Type":"ContainerDied","Data":"7732ddd57e4ba412c592d28e5c00601895b5f9ab906f82b8412755bb317dcf19"} Jul 08 19:18:00.996969 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.996811 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" event={"ID":"e8c49737-2a60-45a5-a337-9cd86ae76e58","Type":"ContainerStarted","Data":"e91c5c8a0b087e809f859036d1bc4fe626fe694c7c1f477c3559a5a075fc1b06"} Jul 08 19:18:00.998562 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.998537 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" event={"ID":"5538bfe3-3461-414b-bc7e-dfef1e582edb","Type":"ContainerStarted","Data":"79317c70667666c54f1879245cda9d31df3cfb530a9f4866906811b2907dfcef"} Jul 08 19:18:00.998677 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:00.998576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" event={"ID":"5538bfe3-3461-414b-bc7e-dfef1e582edb","Type":"ContainerStarted","Data":"bbf9a1aedc7cefd63d1b6d62a8924575f7eefb65080f47f57b55499159bed28f"} Jul 08 19:18:01.000447 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:01.000300 2576 generic.go:356] "Generic (PLEG): container finished" podID="40918d12-8097-4ba3-b1eb-d13ec143f851" containerID="82893842d0a1c8b437bbee798a6b06fc720c8cfef2238f35d4ab3ac4e613d780" exitCode=0 Jul 08 19:18:01.000447 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:01.000378 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" event={"ID":"40918d12-8097-4ba3-b1eb-d13ec143f851","Type":"ContainerDied","Data":"82893842d0a1c8b437bbee798a6b06fc720c8cfef2238f35d4ab3ac4e613d780"} Jul 08 19:18:01.002120 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:01.002100 2576 generic.go:356] "Generic (PLEG): container finished" podID="3317a351-ff9f-4255-a3c4-282552c4c483" containerID="cae28a170dad075a53c30eae5c69cf79c08af76d458fcc71ac75cd1c0ab79248" exitCode=0 Jul 08 19:18:01.002234 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:01.002129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" event={"ID":"3317a351-ff9f-4255-a3c4-282552c4c483","Type":"ContainerDied","Data":"cae28a170dad075a53c30eae5c69cf79c08af76d458fcc71ac75cd1c0ab79248"} Jul 08 19:18:01.002234 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:01.002154 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" event={"ID":"3317a351-ff9f-4255-a3c4-282552c4c483","Type":"ContainerStarted","Data":"acc79dbf4fe35a98dc2573652d68203350823f0f34c00adf31b1fd7adeaed188"} Jul 08 19:18:02.009019 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:02.008944 2576 generic.go:356] "Generic (PLEG): container finished" podID="5538bfe3-3461-414b-bc7e-dfef1e582edb" containerID="79317c70667666c54f1879245cda9d31df3cfb530a9f4866906811b2907dfcef" exitCode=0 Jul 08 19:18:02.009019 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:02.008978 2576 generic.go:356] "Generic (PLEG): container finished" podID="5538bfe3-3461-414b-bc7e-dfef1e582edb" containerID="b1c43c14976e86982a69844101cec18fcf909c6cd79055b222a972919034027b" exitCode=0 Jul 08 19:18:02.009532 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:02.009048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" event={"ID":"5538bfe3-3461-414b-bc7e-dfef1e582edb","Type":"ContainerDied","Data":"79317c70667666c54f1879245cda9d31df3cfb530a9f4866906811b2907dfcef"} Jul 08 19:18:02.009532 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:02.009104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" event={"ID":"5538bfe3-3461-414b-bc7e-dfef1e582edb","Type":"ContainerDied","Data":"b1c43c14976e86982a69844101cec18fcf909c6cd79055b222a972919034027b"} Jul 08 19:18:02.011371 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:02.011336 2576 generic.go:356] "Generic (PLEG): container finished" podID="40918d12-8097-4ba3-b1eb-d13ec143f851" containerID="207a383bcb6d583588f6cec486870a025384271693a103f7396cc2300d093c9b" exitCode=0 Jul 08 19:18:02.011481 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:02.011438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" event={"ID":"40918d12-8097-4ba3-b1eb-d13ec143f851","Type":"ContainerDied","Data":"207a383bcb6d583588f6cec486870a025384271693a103f7396cc2300d093c9b"} Jul 08 19:18:02.013301 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:02.013247 2576 generic.go:356] "Generic (PLEG): container finished" podID="3317a351-ff9f-4255-a3c4-282552c4c483" containerID="333766e62f874f882e3642426fa3e9579be6ba949b91d7589799fc67301f3767" exitCode=0 Jul 08 19:18:02.013415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:02.013301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" event={"ID":"3317a351-ff9f-4255-a3c4-282552c4c483","Type":"ContainerDied","Data":"333766e62f874f882e3642426fa3e9579be6ba949b91d7589799fc67301f3767"} Jul 08 19:18:03.020945 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.020913 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8c49737-2a60-45a5-a337-9cd86ae76e58" containerID="0281db569c124896fe209a0e92b19cfcb8faf67116186b4159fca6e333372ce6" exitCode=0 Jul 08 19:18:03.021412 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.021002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" event={"ID":"e8c49737-2a60-45a5-a337-9cd86ae76e58","Type":"ContainerDied","Data":"0281db569c124896fe209a0e92b19cfcb8faf67116186b4159fca6e333372ce6"} Jul 08 19:18:03.023308 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.023267 2576 generic.go:356] "Generic (PLEG): container finished" podID="5538bfe3-3461-414b-bc7e-dfef1e582edb" containerID="d1a5899c2482b41cb6fa2d4ce220542957efb1c8a33b8080f3a4736cf53d0eb5" exitCode=0 Jul 08 19:18:03.023411 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.023360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" event={"ID":"5538bfe3-3461-414b-bc7e-dfef1e582edb","Type":"ContainerDied","Data":"d1a5899c2482b41cb6fa2d4ce220542957efb1c8a33b8080f3a4736cf53d0eb5"} Jul 08 19:18:03.025582 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.025562 2576 generic.go:356] "Generic (PLEG): container finished" podID="3317a351-ff9f-4255-a3c4-282552c4c483" containerID="44df0fbabe2705af6db53d917585aae5485758e844833e190d47d370d6602f73" exitCode=0 Jul 08 19:18:03.025998 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.025926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" event={"ID":"3317a351-ff9f-4255-a3c4-282552c4c483","Type":"ContainerDied","Data":"44df0fbabe2705af6db53d917585aae5485758e844833e190d47d370d6602f73"} Jul 08 19:18:03.215967 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.215944 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:18:03.276220 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.276158 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wg47\" (UniqueName: \"kubernetes.io/projected/40918d12-8097-4ba3-b1eb-d13ec143f851-kube-api-access-9wg47\") pod \"40918d12-8097-4ba3-b1eb-d13ec143f851\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " Jul 08 19:18:03.276220 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.276194 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-bundle\") pod \"40918d12-8097-4ba3-b1eb-d13ec143f851\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " Jul 08 19:18:03.276397 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.276234 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-util\") pod \"40918d12-8097-4ba3-b1eb-d13ec143f851\" (UID: \"40918d12-8097-4ba3-b1eb-d13ec143f851\") " Jul 08 19:18:03.276758 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.276733 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-bundle" (OuterVolumeSpecName: "bundle") pod "40918d12-8097-4ba3-b1eb-d13ec143f851" (UID: "40918d12-8097-4ba3-b1eb-d13ec143f851"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:18:03.278164 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.278140 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40918d12-8097-4ba3-b1eb-d13ec143f851-kube-api-access-9wg47" (OuterVolumeSpecName: "kube-api-access-9wg47") pod "40918d12-8097-4ba3-b1eb-d13ec143f851" (UID: "40918d12-8097-4ba3-b1eb-d13ec143f851"). InnerVolumeSpecName "kube-api-access-9wg47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:18:03.281315 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.281297 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-util" (OuterVolumeSpecName: "util") pod "40918d12-8097-4ba3-b1eb-d13ec143f851" (UID: "40918d12-8097-4ba3-b1eb-d13ec143f851"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:18:03.376756 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.376729 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-bundle\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:03.376756 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.376753 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40918d12-8097-4ba3-b1eb-d13ec143f851-util\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:03.376756 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:03.376762 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wg47\" (UniqueName: \"kubernetes.io/projected/40918d12-8097-4ba3-b1eb-d13ec143f851-kube-api-access-9wg47\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:04.031683 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.031654 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" Jul 08 19:18:04.032052 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.031696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd" event={"ID":"40918d12-8097-4ba3-b1eb-d13ec143f851","Type":"ContainerDied","Data":"219af02224b61460f68fdfbd74a68df00802ec233ec3026af11538d08d71eda2"} Jul 08 19:18:04.032052 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.031747 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="219af02224b61460f68fdfbd74a68df00802ec233ec3026af11538d08d71eda2" Jul 08 19:18:04.033877 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.033852 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8c49737-2a60-45a5-a337-9cd86ae76e58" containerID="e3410d6dfec1105f722598eaa2aa3da40d76babc36cbad9434d91f427a73627c" exitCode=0 Jul 08 19:18:04.033978 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.033927 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" event={"ID":"e8c49737-2a60-45a5-a337-9cd86ae76e58","Type":"ContainerDied","Data":"e3410d6dfec1105f722598eaa2aa3da40d76babc36cbad9434d91f427a73627c"} Jul 08 19:18:04.169614 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.169589 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:18:04.200093 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.200070 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:04.284198 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.284122 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-bundle\") pod \"3317a351-ff9f-4255-a3c4-282552c4c483\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " Jul 08 19:18:04.284198 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.284154 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xshw4\" (UniqueName: \"kubernetes.io/projected/3317a351-ff9f-4255-a3c4-282552c4c483-kube-api-access-xshw4\") pod \"3317a351-ff9f-4255-a3c4-282552c4c483\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " Jul 08 19:18:04.284198 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.284172 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-util\") pod \"5538bfe3-3461-414b-bc7e-dfef1e582edb\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " Jul 08 19:18:04.284488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.284246 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7md4\" (UniqueName: \"kubernetes.io/projected/5538bfe3-3461-414b-bc7e-dfef1e582edb-kube-api-access-v7md4\") pod \"5538bfe3-3461-414b-bc7e-dfef1e582edb\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " Jul 08 19:18:04.284488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.284275 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-util\") pod \"3317a351-ff9f-4255-a3c4-282552c4c483\" (UID: \"3317a351-ff9f-4255-a3c4-282552c4c483\") " Jul 08 19:18:04.284488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.284300 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-bundle\") pod \"5538bfe3-3461-414b-bc7e-dfef1e582edb\" (UID: \"5538bfe3-3461-414b-bc7e-dfef1e582edb\") " Jul 08 19:18:04.284826 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.284797 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-bundle" (OuterVolumeSpecName: "bundle") pod "3317a351-ff9f-4255-a3c4-282552c4c483" (UID: "3317a351-ff9f-4255-a3c4-282552c4c483"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:18:04.285027 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.285000 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-bundle" (OuterVolumeSpecName: "bundle") pod "5538bfe3-3461-414b-bc7e-dfef1e582edb" (UID: "5538bfe3-3461-414b-bc7e-dfef1e582edb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:18:04.286377 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.286334 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3317a351-ff9f-4255-a3c4-282552c4c483-kube-api-access-xshw4" (OuterVolumeSpecName: "kube-api-access-xshw4") pod "3317a351-ff9f-4255-a3c4-282552c4c483" (UID: "3317a351-ff9f-4255-a3c4-282552c4c483"). InnerVolumeSpecName "kube-api-access-xshw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:18:04.286676 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.286649 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5538bfe3-3461-414b-bc7e-dfef1e582edb-kube-api-access-v7md4" (OuterVolumeSpecName: "kube-api-access-v7md4") pod "5538bfe3-3461-414b-bc7e-dfef1e582edb" (UID: "5538bfe3-3461-414b-bc7e-dfef1e582edb"). InnerVolumeSpecName "kube-api-access-v7md4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:18:04.290509 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.290480 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-util" (OuterVolumeSpecName: "util") pod "5538bfe3-3461-414b-bc7e-dfef1e582edb" (UID: "5538bfe3-3461-414b-bc7e-dfef1e582edb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:18:04.292669 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.292646 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-util" (OuterVolumeSpecName: "util") pod "3317a351-ff9f-4255-a3c4-282552c4c483" (UID: "3317a351-ff9f-4255-a3c4-282552c4c483"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:18:04.385821 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.385792 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-bundle\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:04.385821 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.385817 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xshw4\" (UniqueName: \"kubernetes.io/projected/3317a351-ff9f-4255-a3c4-282552c4c483-kube-api-access-xshw4\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:04.385982 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.385831 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-util\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:04.385982 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.385843 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7md4\" (UniqueName: \"kubernetes.io/projected/5538bfe3-3461-414b-bc7e-dfef1e582edb-kube-api-access-v7md4\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:04.385982 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.385854 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3317a351-ff9f-4255-a3c4-282552c4c483-util\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:04.385982 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:04.385865 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5538bfe3-3461-414b-bc7e-dfef1e582edb-bundle\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:05.040876 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.040850 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" Jul 08 19:18:05.041314 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.040883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8" event={"ID":"5538bfe3-3461-414b-bc7e-dfef1e582edb","Type":"ContainerDied","Data":"bbf9a1aedc7cefd63d1b6d62a8924575f7eefb65080f47f57b55499159bed28f"} Jul 08 19:18:05.041314 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.040939 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbf9a1aedc7cefd63d1b6d62a8924575f7eefb65080f47f57b55499159bed28f" Jul 08 19:18:05.042833 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.042813 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" Jul 08 19:18:05.042958 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.042839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds" event={"ID":"3317a351-ff9f-4255-a3c4-282552c4c483","Type":"ContainerDied","Data":"acc79dbf4fe35a98dc2573652d68203350823f0f34c00adf31b1fd7adeaed188"} Jul 08 19:18:05.042958 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.042874 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acc79dbf4fe35a98dc2573652d68203350823f0f34c00adf31b1fd7adeaed188" Jul 08 19:18:05.175077 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.175055 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:05.294446 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.294378 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-util\") pod \"e8c49737-2a60-45a5-a337-9cd86ae76e58\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " Jul 08 19:18:05.294446 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.294435 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frprq\" (UniqueName: \"kubernetes.io/projected/e8c49737-2a60-45a5-a337-9cd86ae76e58-kube-api-access-frprq\") pod \"e8c49737-2a60-45a5-a337-9cd86ae76e58\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " Jul 08 19:18:05.294661 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.294467 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-bundle\") pod \"e8c49737-2a60-45a5-a337-9cd86ae76e58\" (UID: \"e8c49737-2a60-45a5-a337-9cd86ae76e58\") " Jul 08 19:18:05.294911 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.294888 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-bundle" (OuterVolumeSpecName: "bundle") pod "e8c49737-2a60-45a5-a337-9cd86ae76e58" (UID: "e8c49737-2a60-45a5-a337-9cd86ae76e58"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:18:05.296551 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.296531 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c49737-2a60-45a5-a337-9cd86ae76e58-kube-api-access-frprq" (OuterVolumeSpecName: "kube-api-access-frprq") pod "e8c49737-2a60-45a5-a337-9cd86ae76e58" (UID: "e8c49737-2a60-45a5-a337-9cd86ae76e58"). InnerVolumeSpecName "kube-api-access-frprq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:18:05.300020 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.299981 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-util" (OuterVolumeSpecName: "util") pod "e8c49737-2a60-45a5-a337-9cd86ae76e58" (UID: "e8c49737-2a60-45a5-a337-9cd86ae76e58"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:18:05.395234 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.395209 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-util\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:05.395234 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.395235 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frprq\" (UniqueName: \"kubernetes.io/projected/e8c49737-2a60-45a5-a337-9cd86ae76e58-kube-api-access-frprq\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:05.395391 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:05.395249 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8c49737-2a60-45a5-a337-9cd86ae76e58-bundle\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:06.050638 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:06.050604 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" event={"ID":"e8c49737-2a60-45a5-a337-9cd86ae76e58","Type":"ContainerDied","Data":"e91c5c8a0b087e809f859036d1bc4fe626fe694c7c1f477c3559a5a075fc1b06"} Jul 08 19:18:06.050998 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:06.050655 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e91c5c8a0b087e809f859036d1bc4fe626fe694c7c1f477c3559a5a075fc1b06" Jul 08 19:18:06.050998 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:06.050613 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd" Jul 08 19:18:16.003851 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.003816 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh"] Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004130 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3317a351-ff9f-4255-a3c4-282552c4c483" containerName="pull" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004142 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3317a351-ff9f-4255-a3c4-282552c4c483" containerName="pull" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004154 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5538bfe3-3461-414b-bc7e-dfef1e582edb" containerName="util" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004160 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5538bfe3-3461-414b-bc7e-dfef1e582edb" containerName="util" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004167 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c49737-2a60-45a5-a337-9cd86ae76e58" containerName="pull" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004172 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c49737-2a60-45a5-a337-9cd86ae76e58" containerName="pull" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004178 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40918d12-8097-4ba3-b1eb-d13ec143f851" containerName="extract" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004183 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40918d12-8097-4ba3-b1eb-d13ec143f851" containerName="extract" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004190 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3317a351-ff9f-4255-a3c4-282552c4c483" containerName="util" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004194 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3317a351-ff9f-4255-a3c4-282552c4c483" containerName="util" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004202 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40918d12-8097-4ba3-b1eb-d13ec143f851" containerName="pull" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004207 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40918d12-8097-4ba3-b1eb-d13ec143f851" containerName="pull" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004215 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40918d12-8097-4ba3-b1eb-d13ec143f851" containerName="util" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004220 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40918d12-8097-4ba3-b1eb-d13ec143f851" containerName="util" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004225 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5538bfe3-3461-414b-bc7e-dfef1e582edb" containerName="extract" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004231 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5538bfe3-3461-414b-bc7e-dfef1e582edb" containerName="extract" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004236 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c49737-2a60-45a5-a337-9cd86ae76e58" containerName="extract" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004241 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c49737-2a60-45a5-a337-9cd86ae76e58" containerName="extract" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004248 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3317a351-ff9f-4255-a3c4-282552c4c483" containerName="extract" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004254 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3317a351-ff9f-4255-a3c4-282552c4c483" containerName="extract" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004260 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c49737-2a60-45a5-a337-9cd86ae76e58" containerName="util" Jul 08 19:18:16.004254 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004266 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c49737-2a60-45a5-a337-9cd86ae76e58" containerName="util" Jul 08 19:18:16.005082 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004278 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5538bfe3-3461-414b-bc7e-dfef1e582edb" containerName="pull" Jul 08 19:18:16.005082 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004283 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5538bfe3-3461-414b-bc7e-dfef1e582edb" containerName="pull" Jul 08 19:18:16.005082 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004329 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5538bfe3-3461-414b-bc7e-dfef1e582edb" containerName="extract" Jul 08 19:18:16.005082 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004337 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c49737-2a60-45a5-a337-9cd86ae76e58" containerName="extract" Jul 08 19:18:16.005082 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004360 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3317a351-ff9f-4255-a3c4-282552c4c483" containerName="extract" Jul 08 19:18:16.005082 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.004369 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="40918d12-8097-4ba3-b1eb-d13ec143f851" containerName="extract" Jul 08 19:18:16.011262 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.011243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:16.015433 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.015413 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-s6cqn\"" Jul 08 19:18:16.022776 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.022750 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh"] Jul 08 19:18:16.083798 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.083761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25w5z\" (UniqueName: \"kubernetes.io/projected/daf98812-c21e-443b-8e70-d25ea5a97a85-kube-api-access-25w5z\") pod \"limitador-operator-controller-manager-859d7b78bd-cvndh\" (UID: \"daf98812-c21e-443b-8e70-d25ea5a97a85\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:16.185222 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.185179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25w5z\" (UniqueName: \"kubernetes.io/projected/daf98812-c21e-443b-8e70-d25ea5a97a85-kube-api-access-25w5z\") pod \"limitador-operator-controller-manager-859d7b78bd-cvndh\" (UID: \"daf98812-c21e-443b-8e70-d25ea5a97a85\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:16.194422 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.194394 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25w5z\" (UniqueName: \"kubernetes.io/projected/daf98812-c21e-443b-8e70-d25ea5a97a85-kube-api-access-25w5z\") pod \"limitador-operator-controller-manager-859d7b78bd-cvndh\" (UID: \"daf98812-c21e-443b-8e70-d25ea5a97a85\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:16.321729 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.321689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:16.474045 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:16.474022 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh"] Jul 08 19:18:16.475859 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:18:16.475827 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf98812_c21e_443b_8e70_d25ea5a97a85.slice/crio-ab936c91ab36c1aadfd7695e58c186b4e11489c6645e1805921002645220e3c8 WatchSource:0}: Error finding container ab936c91ab36c1aadfd7695e58c186b4e11489c6645e1805921002645220e3c8: Status 404 returned error can't find the container with id ab936c91ab36c1aadfd7695e58c186b4e11489c6645e1805921002645220e3c8 Jul 08 19:18:17.104817 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:17.104777 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" event={"ID":"daf98812-c21e-443b-8e70-d25ea5a97a85","Type":"ContainerStarted","Data":"ab936c91ab36c1aadfd7695e58c186b4e11489c6645e1805921002645220e3c8"} Jul 08 19:18:19.117108 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:19.117075 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" event={"ID":"daf98812-c21e-443b-8e70-d25ea5a97a85","Type":"ContainerStarted","Data":"48f3559e7eb2569c43cd233b56baffe2372bfb1f4a809d3b6d47cea9a9d343da"} Jul 08 19:18:19.117525 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:19.117188 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:19.136842 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:19.136795 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" podStartSLOduration=2.305225554 podStartE2EDuration="4.136779198s" totalImagesPullingTime="1.831553644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:18:19.135884066 +0000 UTC m=+416.438188235" watchObservedRunningTime="2026-07-08 19:18:19.136779198 +0000 UTC m=+416.439083364" Jul 08 19:18:21.201384 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:21.201333 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv"] Jul 08 19:18:21.205618 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:21.205601 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv" Jul 08 19:18:21.209318 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:21.209292 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-22wzv\"" Jul 08 19:18:21.219209 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:21.219185 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv"] Jul 08 19:18:21.330180 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:21.330147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdnvh\" (UniqueName: \"kubernetes.io/projected/8941f64f-6a00-43c4-8f5d-9315478512bd-kube-api-access-rdnvh\") pod \"authorino-operator-75cc8d5cb7-tk7cv\" (UID: \"8941f64f-6a00-43c4-8f5d-9315478512bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv" Jul 08 19:18:21.430546 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:21.430512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnvh\" (UniqueName: \"kubernetes.io/projected/8941f64f-6a00-43c4-8f5d-9315478512bd-kube-api-access-rdnvh\") pod \"authorino-operator-75cc8d5cb7-tk7cv\" (UID: \"8941f64f-6a00-43c4-8f5d-9315478512bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv" Jul 08 19:18:21.444023 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:21.443992 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnvh\" (UniqueName: \"kubernetes.io/projected/8941f64f-6a00-43c4-8f5d-9315478512bd-kube-api-access-rdnvh\") pod \"authorino-operator-75cc8d5cb7-tk7cv\" (UID: \"8941f64f-6a00-43c4-8f5d-9315478512bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv" Jul 08 19:18:21.515454 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:21.515382 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv" Jul 08 19:18:21.646923 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:21.646896 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv"] Jul 08 19:18:21.648300 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:18:21.648274 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8941f64f_6a00_43c4_8f5d_9315478512bd.slice/crio-a0748a3a1e6acf320be71870959423e32f19730c7101261b73710de7cc0f0d6c WatchSource:0}: Error finding container a0748a3a1e6acf320be71870959423e32f19730c7101261b73710de7cc0f0d6c: Status 404 returned error can't find the container with id a0748a3a1e6acf320be71870959423e32f19730c7101261b73710de7cc0f0d6c Jul 08 19:18:22.133216 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:22.133177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv" event={"ID":"8941f64f-6a00-43c4-8f5d-9315478512bd","Type":"ContainerStarted","Data":"a0748a3a1e6acf320be71870959423e32f19730c7101261b73710de7cc0f0d6c"} Jul 08 19:18:24.146571 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:24.146536 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv" event={"ID":"8941f64f-6a00-43c4-8f5d-9315478512bd","Type":"ContainerStarted","Data":"70f4d2d94aeb984182dbc845fd06cf00fdc1a3fb25ca4ee4b6da47a3a0c5024c"} Jul 08 19:18:24.146969 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:24.146733 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv" Jul 08 19:18:24.185604 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:24.185555 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv" podStartSLOduration=1.065306772 podStartE2EDuration="3.185539034s" totalImagesPullingTime="2.120232262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:18:24.182649162 +0000 UTC m=+421.484953328" watchObservedRunningTime="2026-07-08 19:18:24.185539034 +0000 UTC m=+421.487843201" Jul 08 19:18:26.094357 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.094320 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9"] Jul 08 19:18:26.096634 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.096617 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:26.100661 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.100637 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ztr8n\"" Jul 08 19:18:26.109482 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.109458 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9"] Jul 08 19:18:26.174564 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.174536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfpl\" (UniqueName: \"kubernetes.io/projected/e366c8d4-ec16-4d7a-819d-f7695242c53b-kube-api-access-vrfpl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7r9x9\" (UID: \"e366c8d4-ec16-4d7a-819d-f7695242c53b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:26.174694 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.174577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e366c8d4-ec16-4d7a-819d-f7695242c53b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7r9x9\" (UID: \"e366c8d4-ec16-4d7a-819d-f7695242c53b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:26.275954 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.275915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfpl\" (UniqueName: \"kubernetes.io/projected/e366c8d4-ec16-4d7a-819d-f7695242c53b-kube-api-access-vrfpl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7r9x9\" (UID: \"e366c8d4-ec16-4d7a-819d-f7695242c53b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:26.276138 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.275967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e366c8d4-ec16-4d7a-819d-f7695242c53b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7r9x9\" (UID: \"e366c8d4-ec16-4d7a-819d-f7695242c53b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:26.276340 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.276322 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e366c8d4-ec16-4d7a-819d-f7695242c53b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7r9x9\" (UID: \"e366c8d4-ec16-4d7a-819d-f7695242c53b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:26.286610 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.286590 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfpl\" (UniqueName: \"kubernetes.io/projected/e366c8d4-ec16-4d7a-819d-f7695242c53b-kube-api-access-vrfpl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7r9x9\" (UID: \"e366c8d4-ec16-4d7a-819d-f7695242c53b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:26.406540 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.406455 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:26.566823 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:26.566800 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9"] Jul 08 19:18:26.568468 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:18:26.568446 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode366c8d4_ec16_4d7a_819d_f7695242c53b.slice/crio-91dd6d097fb2c68c74dd34073fd9df2f86a27154104a2537f0027d603da63229 WatchSource:0}: Error finding container 91dd6d097fb2c68c74dd34073fd9df2f86a27154104a2537f0027d603da63229: Status 404 returned error can't find the container with id 91dd6d097fb2c68c74dd34073fd9df2f86a27154104a2537f0027d603da63229 Jul 08 19:18:27.162820 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:27.162784 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" event={"ID":"e366c8d4-ec16-4d7a-819d-f7695242c53b","Type":"ContainerStarted","Data":"91dd6d097fb2c68c74dd34073fd9df2f86a27154104a2537f0027d603da63229"} Jul 08 19:18:30.124398 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:30.124330 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:34.202179 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:34.202132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" event={"ID":"e366c8d4-ec16-4d7a-819d-f7695242c53b","Type":"ContainerStarted","Data":"712ccf4d99911681889a51225209b3d4e95213ff36819a4fc41ec729b9f949cd"} Jul 08 19:18:34.202601 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:34.202230 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:34.225924 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:34.225863 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" podStartSLOduration=1.331600195 podStartE2EDuration="8.225845757s" totalImagesPullingTime="6.894245562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:18:34.222794577 +0000 UTC m=+431.525098744" watchObservedRunningTime="2026-07-08 19:18:34.225845757 +0000 UTC m=+431.528149923" Jul 08 19:18:35.154091 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:35.154058 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tk7cv" Jul 08 19:18:43.340981 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:43.340942 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d495698fb-7dj6j"] Jul 08 19:18:45.210035 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:45.210004 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:47.066077 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.066044 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9"] Jul 08 19:18:47.066537 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.066296 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" podUID="e366c8d4-ec16-4d7a-819d-f7695242c53b" containerName="manager" containerID="cri-o://712ccf4d99911681889a51225209b3d4e95213ff36819a4fc41ec729b9f949cd" gracePeriod=2 Jul 08 19:18:47.074423 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.074391 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9"] Jul 08 19:18:47.093986 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.093952 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh"] Jul 08 19:18:47.094267 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.094219 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" podUID="daf98812-c21e-443b-8e70-d25ea5a97a85" containerName="manager" containerID="cri-o://48f3559e7eb2569c43cd233b56baffe2372bfb1f4a809d3b6d47cea9a9d343da" gracePeriod=2 Jul 08 19:18:47.118266 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.118230 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf"] Jul 08 19:18:47.118676 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.118635 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e366c8d4-ec16-4d7a-819d-f7695242c53b" containerName="manager" Jul 08 19:18:47.118767 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.118681 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e366c8d4-ec16-4d7a-819d-f7695242c53b" containerName="manager" Jul 08 19:18:47.118823 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.118772 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e366c8d4-ec16-4d7a-819d-f7695242c53b" containerName="manager" Jul 08 19:18:47.120895 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.120877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:18:47.125592 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.125568 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh"] Jul 08 19:18:47.127956 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:18:47.127917 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"daf98812-c21e-443b-8e70-d25ea5a97a85\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-cvndh\": pods \"limitador-operator-controller-manager-859d7b78bd-cvndh\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:47.130520 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:18:47.130495 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7r9x9\" is forbidden: User \"system:node:ip-10-0-128-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-188.ec2.internal' and this object" podUID="e366c8d4-ec16-4d7a-819d-f7695242c53b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:47.150239 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.150214 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf"] Jul 08 19:18:47.150447 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.150428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thzjw\" (UniqueName: \"kubernetes.io/projected/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-kube-api-access-thzjw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2qkbf\" (UID: \"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:18:47.150558 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.150543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2qkbf\" (UID: \"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:18:47.154995 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.154965 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz"] Jul 08 19:18:47.155460 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.155443 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daf98812-c21e-443b-8e70-d25ea5a97a85" containerName="manager" Jul 08 19:18:47.155460 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.155460 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="daf98812-c21e-443b-8e70-d25ea5a97a85" containerName="manager" Jul 08 19:18:47.155618 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.155517 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="daf98812-c21e-443b-8e70-d25ea5a97a85" containerName="manager" Jul 08 19:18:47.157769 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.157749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz" Jul 08 19:18:47.174004 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:18:47.173968 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cvndh\" is forbidden: User \"system:node:ip-10-0-128-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-188.ec2.internal' and this object" podUID="daf98812-c21e-443b-8e70-d25ea5a97a85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:47.177602 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.177579 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz"] Jul 08 19:18:47.178670 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:18:47.178637 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7r9x9\" is forbidden: User \"system:node:ip-10-0-128-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-188.ec2.internal' and this object" podUID="e366c8d4-ec16-4d7a-819d-f7695242c53b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:47.181149 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:18:47.181126 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cvndh\" is forbidden: User \"system:node:ip-10-0-128-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-188.ec2.internal' and this object" podUID="daf98812-c21e-443b-8e70-d25ea5a97a85" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:47.184559 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:18:47.184532 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7r9x9\" is forbidden: User \"system:node:ip-10-0-128-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-188.ec2.internal' and this object" podUID="e366c8d4-ec16-4d7a-819d-f7695242c53b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:47.251860 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.251821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thzjw\" (UniqueName: \"kubernetes.io/projected/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-kube-api-access-thzjw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2qkbf\" (UID: \"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:18:47.252163 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.252146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qzkl\" (UniqueName: \"kubernetes.io/projected/79c731d1-ded9-44d4-9b6f-035c1d0c55bc-kube-api-access-5qzkl\") pod \"limitador-operator-controller-manager-859d7b78bd-gc8cz\" (UID: \"79c731d1-ded9-44d4-9b6f-035c1d0c55bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz" Jul 08 19:18:47.252244 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.252233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2qkbf\" (UID: \"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:18:47.252580 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.252564 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2qkbf\" (UID: \"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:18:47.267657 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.267622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thzjw\" (UniqueName: \"kubernetes.io/projected/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-kube-api-access-thzjw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2qkbf\" (UID: \"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:18:47.276141 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.276098 2576 generic.go:356] "Generic (PLEG): container finished" podID="daf98812-c21e-443b-8e70-d25ea5a97a85" containerID="48f3559e7eb2569c43cd233b56baffe2372bfb1f4a809d3b6d47cea9a9d343da" exitCode=0 Jul 08 19:18:47.278569 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.278546 2576 generic.go:356] "Generic (PLEG): container finished" podID="e366c8d4-ec16-4d7a-819d-f7695242c53b" containerID="712ccf4d99911681889a51225209b3d4e95213ff36819a4fc41ec729b9f949cd" exitCode=0 Jul 08 19:18:47.334044 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.334017 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:47.337039 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.337022 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:47.353318 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.353299 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e366c8d4-ec16-4d7a-819d-f7695242c53b-extensions-socket-volume\") pod \"e366c8d4-ec16-4d7a-819d-f7695242c53b\" (UID: \"e366c8d4-ec16-4d7a-819d-f7695242c53b\") " Jul 08 19:18:47.353419 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.353368 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrfpl\" (UniqueName: \"kubernetes.io/projected/e366c8d4-ec16-4d7a-819d-f7695242c53b-kube-api-access-vrfpl\") pod \"e366c8d4-ec16-4d7a-819d-f7695242c53b\" (UID: \"e366c8d4-ec16-4d7a-819d-f7695242c53b\") " Jul 08 19:18:47.353471 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.353445 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25w5z\" (UniqueName: \"kubernetes.io/projected/daf98812-c21e-443b-8e70-d25ea5a97a85-kube-api-access-25w5z\") pod \"daf98812-c21e-443b-8e70-d25ea5a97a85\" (UID: \"daf98812-c21e-443b-8e70-d25ea5a97a85\") " Jul 08 19:18:47.353605 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.353584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qzkl\" (UniqueName: \"kubernetes.io/projected/79c731d1-ded9-44d4-9b6f-035c1d0c55bc-kube-api-access-5qzkl\") pod \"limitador-operator-controller-manager-859d7b78bd-gc8cz\" (UID: \"79c731d1-ded9-44d4-9b6f-035c1d0c55bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz" Jul 08 19:18:47.353795 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.353771 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e366c8d4-ec16-4d7a-819d-f7695242c53b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e366c8d4-ec16-4d7a-819d-f7695242c53b" (UID: "e366c8d4-ec16-4d7a-819d-f7695242c53b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:18:47.355445 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.355418 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e366c8d4-ec16-4d7a-819d-f7695242c53b-kube-api-access-vrfpl" (OuterVolumeSpecName: "kube-api-access-vrfpl") pod "e366c8d4-ec16-4d7a-819d-f7695242c53b" (UID: "e366c8d4-ec16-4d7a-819d-f7695242c53b"). InnerVolumeSpecName "kube-api-access-vrfpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:18:47.355556 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.355540 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf98812-c21e-443b-8e70-d25ea5a97a85-kube-api-access-25w5z" (OuterVolumeSpecName: "kube-api-access-25w5z") pod "daf98812-c21e-443b-8e70-d25ea5a97a85" (UID: "daf98812-c21e-443b-8e70-d25ea5a97a85"). InnerVolumeSpecName "kube-api-access-25w5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:18:47.374192 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.374166 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qzkl\" (UniqueName: \"kubernetes.io/projected/79c731d1-ded9-44d4-9b6f-035c1d0c55bc-kube-api-access-5qzkl\") pod \"limitador-operator-controller-manager-859d7b78bd-gc8cz\" (UID: \"79c731d1-ded9-44d4-9b6f-035c1d0c55bc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz" Jul 08 19:18:47.454365 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.454303 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25w5z\" (UniqueName: \"kubernetes.io/projected/daf98812-c21e-443b-8e70-d25ea5a97a85-kube-api-access-25w5z\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:47.454365 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.454365 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e366c8d4-ec16-4d7a-819d-f7695242c53b-extensions-socket-volume\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:47.454519 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.454381 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrfpl\" (UniqueName: \"kubernetes.io/projected/e366c8d4-ec16-4d7a-819d-f7695242c53b-kube-api-access-vrfpl\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:18:47.489799 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.489773 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:18:47.495760 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.495736 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz" Jul 08 19:18:47.640117 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.640090 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf"] Jul 08 19:18:47.642985 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:18:47.642956 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0769eb_dd8b_4ec4_8637_0eeac41a98fb.slice/crio-fdc7f0ce26c27b5eb738895bdbc6e13ad08bcb5f57c633f6313b49f7f2d07df1 WatchSource:0}: Error finding container fdc7f0ce26c27b5eb738895bdbc6e13ad08bcb5f57c633f6313b49f7f2d07df1: Status 404 returned error can't find the container with id fdc7f0ce26c27b5eb738895bdbc6e13ad08bcb5f57c633f6313b49f7f2d07df1 Jul 08 19:18:47.656515 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:47.656491 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz"] Jul 08 19:18:47.658516 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:18:47.658493 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c731d1_ded9_44d4_9b6f_035c1d0c55bc.slice/crio-18e466a1b3d137b48176eaef6175e97d519c77dfa6e111e3e4c789be18350c9a WatchSource:0}: Error finding container 18e466a1b3d137b48176eaef6175e97d519c77dfa6e111e3e4c789be18350c9a: Status 404 returned error can't find the container with id 18e466a1b3d137b48176eaef6175e97d519c77dfa6e111e3e4c789be18350c9a Jul 08 19:18:48.285675 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.285644 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cvndh" Jul 08 19:18:48.286199 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.285654 2576 scope.go:117] "RemoveContainer" containerID="48f3559e7eb2569c43cd233b56baffe2372bfb1f4a809d3b6d47cea9a9d343da" Jul 08 19:18:48.287417 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.287384 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz" event={"ID":"79c731d1-ded9-44d4-9b6f-035c1d0c55bc","Type":"ContainerStarted","Data":"0179488394a4247bd14d441fa9524511ee6f97a929a40efcf3091d3b55234578"} Jul 08 19:18:48.287552 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.287446 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz" event={"ID":"79c731d1-ded9-44d4-9b6f-035c1d0c55bc","Type":"ContainerStarted","Data":"18e466a1b3d137b48176eaef6175e97d519c77dfa6e111e3e4c789be18350c9a"} Jul 08 19:18:48.287552 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.287491 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz" Jul 08 19:18:48.288894 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.288873 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7r9x9" Jul 08 19:18:48.290631 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.290603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" event={"ID":"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb","Type":"ContainerStarted","Data":"6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3"} Jul 08 19:18:48.290748 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.290651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" event={"ID":"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb","Type":"ContainerStarted","Data":"fdc7f0ce26c27b5eb738895bdbc6e13ad08bcb5f57c633f6313b49f7f2d07df1"} Jul 08 19:18:48.290748 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.290728 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:18:48.295103 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.295083 2576 scope.go:117] "RemoveContainer" containerID="712ccf4d99911681889a51225209b3d4e95213ff36819a4fc41ec729b9f949cd" Jul 08 19:18:48.305316 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.305275 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz" podStartSLOduration=1.3052620670000001 podStartE2EDuration="1.305262067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:18:48.304553007 +0000 UTC m=+445.606857174" watchObservedRunningTime="2026-07-08 19:18:48.305262067 +0000 UTC m=+445.607566232" Jul 08 19:18:48.328579 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:48.328531 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" podStartSLOduration=1.328516263 podStartE2EDuration="1.328516263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:18:48.326018412 +0000 UTC m=+445.628322580" watchObservedRunningTime="2026-07-08 19:18:48.328516263 +0000 UTC m=+445.630820430" Jul 08 19:18:49.269444 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:49.269413 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daf98812-c21e-443b-8e70-d25ea5a97a85" path="/var/lib/kubelet/pods/daf98812-c21e-443b-8e70-d25ea5a97a85/volumes" Jul 08 19:18:49.269767 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:49.269755 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e366c8d4-ec16-4d7a-819d-f7695242c53b" path="/var/lib/kubelet/pods/e366c8d4-ec16-4d7a-819d-f7695242c53b/volumes" Jul 08 19:18:59.304772 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:59.304744 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:18:59.305225 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:18:59.304813 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gc8cz" Jul 08 19:19:08.361966 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.361920 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7d495698fb-7dj6j" podUID="1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" containerName="console" containerID="cri-o://099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e" gracePeriod=15 Jul 08 19:19:08.599245 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.599222 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:19:08.722727 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.722642 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-service-ca\") pod \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " Jul 08 19:19:08.722727 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.722704 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-config\") pod \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " Jul 08 19:19:08.722979 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.722814 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-trusted-ca-bundle\") pod \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " Jul 08 19:19:08.722979 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.722861 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-oauth-serving-cert\") pod \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " Jul 08 19:19:08.722979 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.722880 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-oauth-config\") pod \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " Jul 08 19:19:08.722979 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.722896 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhsdw\" (UniqueName: \"kubernetes.io/projected/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-kube-api-access-xhsdw\") pod \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " Jul 08 19:19:08.722979 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.722919 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-serving-cert\") pod \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\" (UID: \"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1\") " Jul 08 19:19:08.723241 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.723151 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-config" (OuterVolumeSpecName: "console-config") pod "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" (UID: "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:19:08.723241 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.723153 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-service-ca" (OuterVolumeSpecName: "service-ca") pod "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" (UID: "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:19:08.723336 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.723274 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" (UID: "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:19:08.723336 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.723297 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" (UID: "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:19:08.725068 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.725041 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" (UID: "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:19:08.725068 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.725057 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" (UID: "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:19:08.725216 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.725053 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-kube-api-access-xhsdw" (OuterVolumeSpecName: "kube-api-access-xhsdw") pod "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" (UID: "1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1"). InnerVolumeSpecName "kube-api-access-xhsdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:19:08.824214 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.824178 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-config\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:08.824214 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.824205 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-trusted-ca-bundle\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:08.824214 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.824215 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-oauth-serving-cert\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:08.824214 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.824224 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-oauth-config\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:08.824518 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.824234 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhsdw\" (UniqueName: \"kubernetes.io/projected/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-kube-api-access-xhsdw\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:08.824518 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.824242 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-console-serving-cert\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:08.824518 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:08.824252 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1-service-ca\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:09.402520 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:09.402486 2576 generic.go:356] "Generic (PLEG): container finished" podID="1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" containerID="099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e" exitCode=0 Jul 08 19:19:09.402928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:09.402549 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d495698fb-7dj6j" Jul 08 19:19:09.402928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:09.402567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d495698fb-7dj6j" event={"ID":"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1","Type":"ContainerDied","Data":"099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e"} Jul 08 19:19:09.402928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:09.402623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d495698fb-7dj6j" event={"ID":"1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1","Type":"ContainerDied","Data":"2e1ebfe8f1960c32030c5b375675e3822dbfcbe69b63dd0424ed96c535c60e09"} Jul 08 19:19:09.402928 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:09.402650 2576 scope.go:117] "RemoveContainer" containerID="099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e" Jul 08 19:19:09.411715 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:09.411695 2576 scope.go:117] "RemoveContainer" containerID="099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e" Jul 08 19:19:09.411974 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:19:09.411957 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e\": container with ID starting with 099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e not found: ID does not exist" containerID="099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e" Jul 08 19:19:09.412020 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:09.411984 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e"} err="failed to get container status \"099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e\": rpc error: code = NotFound desc = could not find container \"099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e\": container with ID starting with 099e06664222afb09d41633cf37bf80b84236b855aa21c2f2e6157fd3fdd1e6e not found: ID does not exist" Jul 08 19:19:09.420137 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:09.420115 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d495698fb-7dj6j"] Jul 08 19:19:09.426693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:09.426670 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d495698fb-7dj6j"] Jul 08 19:19:11.268109 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:11.268074 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" path="/var/lib/kubelet/pods/1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1/volumes" Jul 08 19:19:23.510087 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.510052 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf"] Jul 08 19:19:23.510487 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.510259 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" podUID="6f0769eb-dd8b-4ec4-8637-0eeac41a98fb" containerName="manager" containerID="cri-o://6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3" gracePeriod=10 Jul 08 19:19:23.770902 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.770881 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:19:23.777298 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.777275 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7"] Jul 08 19:19:23.777668 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.777650 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f0769eb-dd8b-4ec4-8637-0eeac41a98fb" containerName="manager" Jul 08 19:19:23.777754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.777671 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f0769eb-dd8b-4ec4-8637-0eeac41a98fb" containerName="manager" Jul 08 19:19:23.777754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.777695 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" containerName="console" Jul 08 19:19:23.777754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.777703 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" containerName="console" Jul 08 19:19:23.777854 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.777804 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d614a45-cc3e-4e9a-85e2-4cbf890ca7b1" containerName="console" Jul 08 19:19:23.777854 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.777822 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f0769eb-dd8b-4ec4-8637-0eeac41a98fb" containerName="manager" Jul 08 19:19:23.779693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.779675 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:19:23.796991 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.796971 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7"] Jul 08 19:19:23.851595 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.851562 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thzjw\" (UniqueName: \"kubernetes.io/projected/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-kube-api-access-thzjw\") pod \"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb\" (UID: \"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb\") " Jul 08 19:19:23.851754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.851607 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-extensions-socket-volume\") pod \"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb\" (UID: \"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb\") " Jul 08 19:19:23.851754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.851723 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swtxf\" (UniqueName: \"kubernetes.io/projected/65b32cfb-564f-4e7a-981a-f724d35c1716-kube-api-access-swtxf\") pod \"kuadrant-operator-controller-manager-86878c769f-z62d7\" (UID: \"65b32cfb-564f-4e7a-981a-f724d35c1716\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:19:23.851754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.851749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65b32cfb-564f-4e7a-981a-f724d35c1716-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z62d7\" (UID: \"65b32cfb-564f-4e7a-981a-f724d35c1716\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:19:23.852027 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.852005 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6f0769eb-dd8b-4ec4-8637-0eeac41a98fb" (UID: "6f0769eb-dd8b-4ec4-8637-0eeac41a98fb"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:19:23.853741 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.853718 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-kube-api-access-thzjw" (OuterVolumeSpecName: "kube-api-access-thzjw") pod "6f0769eb-dd8b-4ec4-8637-0eeac41a98fb" (UID: "6f0769eb-dd8b-4ec4-8637-0eeac41a98fb"). InnerVolumeSpecName "kube-api-access-thzjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:19:23.952486 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.952451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swtxf\" (UniqueName: \"kubernetes.io/projected/65b32cfb-564f-4e7a-981a-f724d35c1716-kube-api-access-swtxf\") pod \"kuadrant-operator-controller-manager-86878c769f-z62d7\" (UID: \"65b32cfb-564f-4e7a-981a-f724d35c1716\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:19:23.952486 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.952492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65b32cfb-564f-4e7a-981a-f724d35c1716-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z62d7\" (UID: \"65b32cfb-564f-4e7a-981a-f724d35c1716\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:19:23.952677 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.952535 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thzjw\" (UniqueName: \"kubernetes.io/projected/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-kube-api-access-thzjw\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:23.952677 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.952545 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb-extensions-socket-volume\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:23.952830 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.952816 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65b32cfb-564f-4e7a-981a-f724d35c1716-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z62d7\" (UID: \"65b32cfb-564f-4e7a-981a-f724d35c1716\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:19:23.963365 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:23.963323 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swtxf\" (UniqueName: \"kubernetes.io/projected/65b32cfb-564f-4e7a-981a-f724d35c1716-kube-api-access-swtxf\") pod \"kuadrant-operator-controller-manager-86878c769f-z62d7\" (UID: \"65b32cfb-564f-4e7a-981a-f724d35c1716\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:19:24.089305 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.089272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:19:24.228167 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.228130 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7"] Jul 08 19:19:24.231530 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:19:24.231494 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b32cfb_564f_4e7a_981a_f724d35c1716.slice/crio-e34504d74db8e2498caeb52a4e3179d3231b745793aac6e406b60802b625a7cd WatchSource:0}: Error finding container e34504d74db8e2498caeb52a4e3179d3231b745793aac6e406b60802b625a7cd: Status 404 returned error can't find the container with id e34504d74db8e2498caeb52a4e3179d3231b745793aac6e406b60802b625a7cd Jul 08 19:19:24.478710 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.478621 2576 generic.go:356] "Generic (PLEG): container finished" podID="6f0769eb-dd8b-4ec4-8637-0eeac41a98fb" containerID="6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3" exitCode=0 Jul 08 19:19:24.478710 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.478670 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" Jul 08 19:19:24.478903 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.478712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" event={"ID":"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb","Type":"ContainerDied","Data":"6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3"} Jul 08 19:19:24.478903 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.478767 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf" event={"ID":"6f0769eb-dd8b-4ec4-8637-0eeac41a98fb","Type":"ContainerDied","Data":"fdc7f0ce26c27b5eb738895bdbc6e13ad08bcb5f57c633f6313b49f7f2d07df1"} Jul 08 19:19:24.478903 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.478794 2576 scope.go:117] "RemoveContainer" containerID="6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3" Jul 08 19:19:24.480393 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.480369 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" event={"ID":"65b32cfb-564f-4e7a-981a-f724d35c1716","Type":"ContainerStarted","Data":"62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a"} Jul 08 19:19:24.480487 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.480410 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" event={"ID":"65b32cfb-564f-4e7a-981a-f724d35c1716","Type":"ContainerStarted","Data":"e34504d74db8e2498caeb52a4e3179d3231b745793aac6e406b60802b625a7cd"} Jul 08 19:19:24.480543 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.480533 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:19:24.488092 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.488078 2576 scope.go:117] "RemoveContainer" containerID="6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3" Jul 08 19:19:24.488339 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:19:24.488322 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3\": container with ID starting with 6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3 not found: ID does not exist" containerID="6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3" Jul 08 19:19:24.488339 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.488362 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3"} err="failed to get container status \"6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3\": rpc error: code = NotFound desc = could not find container \"6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3\": container with ID starting with 6bd3f2a914572aa171d5bd6fcd9ff4e0fb1f0ada92570946e4fc8ef4291e0ca3 not found: ID does not exist" Jul 08 19:19:24.501507 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.501467 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" podStartSLOduration=1.5014558089999999 podStartE2EDuration="1.501455809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:19:24.500117869 +0000 UTC m=+481.802422037" watchObservedRunningTime="2026-07-08 19:19:24.501455809 +0000 UTC m=+481.803759976" Jul 08 19:19:24.518518 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.518491 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf"] Jul 08 19:19:24.522579 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:24.522557 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2qkbf"] Jul 08 19:19:25.268228 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:25.268194 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f0769eb-dd8b-4ec4-8637-0eeac41a98fb" path="/var/lib/kubelet/pods/6f0769eb-dd8b-4ec4-8637-0eeac41a98fb/volumes" Jul 08 19:19:35.489127 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:35.489095 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:19:42.471232 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.471199 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jd8dl"] Jul 08 19:19:42.474866 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.474843 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:42.477838 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.477806 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kk9wf\"" Jul 08 19:19:42.477838 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.477819 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 19:19:42.483734 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.483711 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jd8dl"] Jul 08 19:19:42.511639 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.511610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0b86504-c726-442d-893b-4db36ebf968f-config-file\") pod \"limitador-limitador-64c94fd64f-jd8dl\" (UID: \"c0b86504-c726-442d-893b-4db36ebf968f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:42.511754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.511643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrc6\" (UniqueName: \"kubernetes.io/projected/c0b86504-c726-442d-893b-4db36ebf968f-kube-api-access-qsrc6\") pod \"limitador-limitador-64c94fd64f-jd8dl\" (UID: \"c0b86504-c726-442d-893b-4db36ebf968f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:42.572210 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.572179 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jd8dl"] Jul 08 19:19:42.612577 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.612550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0b86504-c726-442d-893b-4db36ebf968f-config-file\") pod \"limitador-limitador-64c94fd64f-jd8dl\" (UID: \"c0b86504-c726-442d-893b-4db36ebf968f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:42.612577 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.612582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrc6\" (UniqueName: \"kubernetes.io/projected/c0b86504-c726-442d-893b-4db36ebf968f-kube-api-access-qsrc6\") pod \"limitador-limitador-64c94fd64f-jd8dl\" (UID: \"c0b86504-c726-442d-893b-4db36ebf968f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:42.613198 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.613179 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0b86504-c726-442d-893b-4db36ebf968f-config-file\") pod \"limitador-limitador-64c94fd64f-jd8dl\" (UID: \"c0b86504-c726-442d-893b-4db36ebf968f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:42.625504 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.625478 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrc6\" (UniqueName: \"kubernetes.io/projected/c0b86504-c726-442d-893b-4db36ebf968f-kube-api-access-qsrc6\") pod \"limitador-limitador-64c94fd64f-jd8dl\" (UID: \"c0b86504-c726-442d-893b-4db36ebf968f\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:42.786948 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.786916 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:42.921495 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:42.921473 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jd8dl"] Jul 08 19:19:42.923642 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:19:42.923612 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b86504_c726_442d_893b_4db36ebf968f.slice/crio-74d0f111e3e5e5c4ab6a02733903094f2bcf923d296defad75f99f2414e8dba3 WatchSource:0}: Error finding container 74d0f111e3e5e5c4ab6a02733903094f2bcf923d296defad75f99f2414e8dba3: Status 404 returned error can't find the container with id 74d0f111e3e5e5c4ab6a02733903094f2bcf923d296defad75f99f2414e8dba3 Jul 08 19:19:43.268818 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.268743 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-4d7kx"] Jul 08 19:19:43.274777 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.274747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4d7kx" Jul 08 19:19:43.276668 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.276642 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-4d7kx"] Jul 08 19:19:43.277523 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.277501 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x89np\"" Jul 08 19:19:43.388714 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.388680 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-tth58"] Jul 08 19:19:43.390962 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.390948 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tth58" Jul 08 19:19:43.398667 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.398620 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tth58"] Jul 08 19:19:43.418200 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.418169 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrc9\" (UniqueName: \"kubernetes.io/projected/3bf6fc20-64b7-43db-9132-3612c6036872-kube-api-access-ggrc9\") pod \"authorino-75d556bd77-tth58\" (UID: \"3bf6fc20-64b7-43db-9132-3612c6036872\") " pod="kuadrant-system/authorino-75d556bd77-tth58" Jul 08 19:19:43.418385 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.418217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklk9\" (UniqueName: \"kubernetes.io/projected/abc7c6e2-46dc-41c4-adbe-b66404fff870-kube-api-access-tklk9\") pod \"authorino-58cd955675-4d7kx\" (UID: \"abc7c6e2-46dc-41c4-adbe-b66404fff870\") " pod="kuadrant-system/authorino-58cd955675-4d7kx" Jul 08 19:19:43.518802 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.518764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrc9\" (UniqueName: \"kubernetes.io/projected/3bf6fc20-64b7-43db-9132-3612c6036872-kube-api-access-ggrc9\") pod \"authorino-75d556bd77-tth58\" (UID: \"3bf6fc20-64b7-43db-9132-3612c6036872\") " pod="kuadrant-system/authorino-75d556bd77-tth58" Jul 08 19:19:43.518802 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.518808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklk9\" (UniqueName: \"kubernetes.io/projected/abc7c6e2-46dc-41c4-adbe-b66404fff870-kube-api-access-tklk9\") pod \"authorino-58cd955675-4d7kx\" (UID: \"abc7c6e2-46dc-41c4-adbe-b66404fff870\") " pod="kuadrant-system/authorino-58cd955675-4d7kx" Jul 08 19:19:43.527399 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.527318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrc9\" (UniqueName: \"kubernetes.io/projected/3bf6fc20-64b7-43db-9132-3612c6036872-kube-api-access-ggrc9\") pod \"authorino-75d556bd77-tth58\" (UID: \"3bf6fc20-64b7-43db-9132-3612c6036872\") " pod="kuadrant-system/authorino-75d556bd77-tth58" Jul 08 19:19:43.527545 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.527405 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklk9\" (UniqueName: \"kubernetes.io/projected/abc7c6e2-46dc-41c4-adbe-b66404fff870-kube-api-access-tklk9\") pod \"authorino-58cd955675-4d7kx\" (UID: \"abc7c6e2-46dc-41c4-adbe-b66404fff870\") " pod="kuadrant-system/authorino-58cd955675-4d7kx" Jul 08 19:19:43.583764 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.583730 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" event={"ID":"c0b86504-c726-442d-893b-4db36ebf968f","Type":"ContainerStarted","Data":"74d0f111e3e5e5c4ab6a02733903094f2bcf923d296defad75f99f2414e8dba3"} Jul 08 19:19:43.585095 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.585078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4d7kx" Jul 08 19:19:43.701878 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.701805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tth58" Jul 08 19:19:43.732115 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.732082 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-4d7kx"] Jul 08 19:19:43.740517 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:19:43.740483 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabc7c6e2_46dc_41c4_adbe_b66404fff870.slice/crio-1c9077cf6bb849d6512625e3105ab9081e875a7eea7fbcb1057f6f26e6d7e2ab WatchSource:0}: Error finding container 1c9077cf6bb849d6512625e3105ab9081e875a7eea7fbcb1057f6f26e6d7e2ab: Status 404 returned error can't find the container with id 1c9077cf6bb849d6512625e3105ab9081e875a7eea7fbcb1057f6f26e6d7e2ab Jul 08 19:19:43.885608 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:43.885563 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tth58"] Jul 08 19:19:44.594399 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:44.594336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tth58" event={"ID":"3bf6fc20-64b7-43db-9132-3612c6036872","Type":"ContainerStarted","Data":"6895aed4109c28fe3ba0141c43abeca008a26b93fc7b366e5eac1bc2871745b8"} Jul 08 19:19:44.596391 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:44.596337 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4d7kx" event={"ID":"abc7c6e2-46dc-41c4-adbe-b66404fff870","Type":"ContainerStarted","Data":"1c9077cf6bb849d6512625e3105ab9081e875a7eea7fbcb1057f6f26e6d7e2ab"} Jul 08 19:19:48.631141 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:48.631101 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" event={"ID":"c0b86504-c726-442d-893b-4db36ebf968f","Type":"ContainerStarted","Data":"4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2"} Jul 08 19:19:48.631606 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:48.631190 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:48.632668 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:48.632639 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4d7kx" event={"ID":"abc7c6e2-46dc-41c4-adbe-b66404fff870","Type":"ContainerStarted","Data":"b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0"} Jul 08 19:19:48.633902 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:48.633877 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tth58" event={"ID":"3bf6fc20-64b7-43db-9132-3612c6036872","Type":"ContainerStarted","Data":"1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544"} Jul 08 19:19:48.649126 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:48.649079 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" podStartSLOduration=1.629488785 podStartE2EDuration="6.649064898s" totalImagesPullingTime="5.019576113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:19:48.647423426 +0000 UTC m=+505.949727594" watchObservedRunningTime="2026-07-08 19:19:48.649064898 +0000 UTC m=+505.951369075" Jul 08 19:19:48.663013 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:48.662969 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-4d7kx" podStartSLOduration=1.418549821 podStartE2EDuration="5.662956995s" totalImagesPullingTime="4.244407174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:19:48.662650147 +0000 UTC m=+505.964954314" watchObservedRunningTime="2026-07-08 19:19:48.662956995 +0000 UTC m=+505.965261161" Jul 08 19:19:48.678838 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:48.678801 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-tth58" podStartSLOduration=1.57899963 podStartE2EDuration="5.67878878s" totalImagesPullingTime="4.09978915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:19:48.678140953 +0000 UTC m=+505.980445122" watchObservedRunningTime="2026-07-08 19:19:48.67878878 +0000 UTC m=+505.981092947" Jul 08 19:19:48.712291 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:48.712258 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-4d7kx"] Jul 08 19:19:50.650427 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:50.650369 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-4d7kx" podUID="abc7c6e2-46dc-41c4-adbe-b66404fff870" containerName="authorino" containerID="cri-o://b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0" gracePeriod=30 Jul 08 19:19:50.899426 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:50.899397 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4d7kx" Jul 08 19:19:50.982429 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:50.982362 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tklk9\" (UniqueName: \"kubernetes.io/projected/abc7c6e2-46dc-41c4-adbe-b66404fff870-kube-api-access-tklk9\") pod \"abc7c6e2-46dc-41c4-adbe-b66404fff870\" (UID: \"abc7c6e2-46dc-41c4-adbe-b66404fff870\") " Jul 08 19:19:50.984390 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:50.984339 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc7c6e2-46dc-41c4-adbe-b66404fff870-kube-api-access-tklk9" (OuterVolumeSpecName: "kube-api-access-tklk9") pod "abc7c6e2-46dc-41c4-adbe-b66404fff870" (UID: "abc7c6e2-46dc-41c4-adbe-b66404fff870"). InnerVolumeSpecName "kube-api-access-tklk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:19:51.083114 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:51.083081 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tklk9\" (UniqueName: \"kubernetes.io/projected/abc7c6e2-46dc-41c4-adbe-b66404fff870-kube-api-access-tklk9\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:51.657300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:51.657270 2576 generic.go:356] "Generic (PLEG): container finished" podID="abc7c6e2-46dc-41c4-adbe-b66404fff870" containerID="b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0" exitCode=0 Jul 08 19:19:51.657708 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:51.657315 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4d7kx" Jul 08 19:19:51.657708 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:51.657381 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4d7kx" event={"ID":"abc7c6e2-46dc-41c4-adbe-b66404fff870","Type":"ContainerDied","Data":"b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0"} Jul 08 19:19:51.657708 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:51.657424 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4d7kx" event={"ID":"abc7c6e2-46dc-41c4-adbe-b66404fff870","Type":"ContainerDied","Data":"1c9077cf6bb849d6512625e3105ab9081e875a7eea7fbcb1057f6f26e6d7e2ab"} Jul 08 19:19:51.657708 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:51.657448 2576 scope.go:117] "RemoveContainer" containerID="b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0" Jul 08 19:19:51.665919 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:51.665904 2576 scope.go:117] "RemoveContainer" containerID="b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0" Jul 08 19:19:51.666167 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:19:51.666147 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0\": container with ID starting with b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0 not found: ID does not exist" containerID="b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0" Jul 08 19:19:51.666213 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:51.666176 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0"} err="failed to get container status \"b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0\": rpc error: code = NotFound desc = could not find container \"b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0\": container with ID starting with b2fe75cc31ec174c2a26462110d57c9281103c43a7616905d64c6252c7848ba0 not found: ID does not exist" Jul 08 19:19:51.675112 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:51.675089 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-4d7kx"] Jul 08 19:19:51.678447 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:51.678427 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-4d7kx"] Jul 08 19:19:53.268190 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:53.268154 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc7c6e2-46dc-41c4-adbe-b66404fff870" path="/var/lib/kubelet/pods/abc7c6e2-46dc-41c4-adbe-b66404fff870/volumes" Jul 08 19:19:57.655644 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:57.655612 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jd8dl"] Jul 08 19:19:57.656070 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:57.655816 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" podUID="c0b86504-c726-442d-893b-4db36ebf968f" containerName="limitador" containerID="cri-o://4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2" gracePeriod=30 Jul 08 19:19:57.658195 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:57.658171 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:58.602831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.602807 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:58.640382 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.640339 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsrc6\" (UniqueName: \"kubernetes.io/projected/c0b86504-c726-442d-893b-4db36ebf968f-kube-api-access-qsrc6\") pod \"c0b86504-c726-442d-893b-4db36ebf968f\" (UID: \"c0b86504-c726-442d-893b-4db36ebf968f\") " Jul 08 19:19:58.640533 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.640494 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0b86504-c726-442d-893b-4db36ebf968f-config-file\") pod \"c0b86504-c726-442d-893b-4db36ebf968f\" (UID: \"c0b86504-c726-442d-893b-4db36ebf968f\") " Jul 08 19:19:58.640834 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.640803 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b86504-c726-442d-893b-4db36ebf968f-config-file" (OuterVolumeSpecName: "config-file") pod "c0b86504-c726-442d-893b-4db36ebf968f" (UID: "c0b86504-c726-442d-893b-4db36ebf968f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:19:58.642378 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.642355 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b86504-c726-442d-893b-4db36ebf968f-kube-api-access-qsrc6" (OuterVolumeSpecName: "kube-api-access-qsrc6") pod "c0b86504-c726-442d-893b-4db36ebf968f" (UID: "c0b86504-c726-442d-893b-4db36ebf968f"). InnerVolumeSpecName "kube-api-access-qsrc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:19:58.693562 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.693531 2576 generic.go:356] "Generic (PLEG): container finished" podID="c0b86504-c726-442d-893b-4db36ebf968f" containerID="4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2" exitCode=0 Jul 08 19:19:58.693967 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.693579 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" Jul 08 19:19:58.693967 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.693585 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" event={"ID":"c0b86504-c726-442d-893b-4db36ebf968f","Type":"ContainerDied","Data":"4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2"} Jul 08 19:19:58.693967 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.693631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jd8dl" event={"ID":"c0b86504-c726-442d-893b-4db36ebf968f","Type":"ContainerDied","Data":"74d0f111e3e5e5c4ab6a02733903094f2bcf923d296defad75f99f2414e8dba3"} Jul 08 19:19:58.693967 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.693658 2576 scope.go:117] "RemoveContainer" containerID="4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2" Jul 08 19:19:58.702549 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.702532 2576 scope.go:117] "RemoveContainer" containerID="4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2" Jul 08 19:19:58.702804 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:19:58.702784 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2\": container with ID starting with 4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2 not found: ID does not exist" containerID="4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2" Jul 08 19:19:58.702858 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.702814 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2"} err="failed to get container status \"4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2\": rpc error: code = NotFound desc = could not find container \"4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2\": container with ID starting with 4c28d5dc7ea6eb93872e1bc6d366fe992acf77cd686881a2d3598a9da4e0bea2 not found: ID does not exist" Jul 08 19:19:58.715036 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.715014 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jd8dl"] Jul 08 19:19:58.718576 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.718529 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jd8dl"] Jul 08 19:19:58.741042 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.741016 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c0b86504-c726-442d-893b-4db36ebf968f-config-file\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:58.741042 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:58.741038 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsrc6\" (UniqueName: \"kubernetes.io/projected/c0b86504-c726-442d-893b-4db36ebf968f-kube-api-access-qsrc6\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:19:59.267992 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:19:59.267951 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b86504-c726-442d-893b-4db36ebf968f" path="/var/lib/kubelet/pods/c0b86504-c726-442d-893b-4db36ebf968f/volumes" Jul 08 19:20:03.798818 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.798786 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6"] Jul 08 19:20:03.799328 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.799310 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc7c6e2-46dc-41c4-adbe-b66404fff870" containerName="authorino" Jul 08 19:20:03.799328 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.799330 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc7c6e2-46dc-41c4-adbe-b66404fff870" containerName="authorino" Jul 08 19:20:03.799516 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.799398 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b86504-c726-442d-893b-4db36ebf968f" containerName="limitador" Jul 08 19:20:03.799516 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.799407 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b86504-c726-442d-893b-4db36ebf968f" containerName="limitador" Jul 08 19:20:03.799516 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.799528 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b86504-c726-442d-893b-4db36ebf968f" containerName="limitador" Jul 08 19:20:03.799516 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.799542 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc7c6e2-46dc-41c4-adbe-b66404fff870" containerName="authorino" Jul 08 19:20:03.802281 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.802260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" Jul 08 19:20:03.805267 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.805243 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 08 19:20:03.805388 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.805249 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 19:20:03.805388 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.805293 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 19:20:03.805388 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.805302 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jz2nm\"" Jul 08 19:20:03.811377 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.811339 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6"] Jul 08 19:20:03.883934 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.883899 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5636bf60-dc82-4f78-b57b-ce8a9211bd02-data\") pod \"postgres-69fb4594cf-c8qq6\" (UID: \"5636bf60-dc82-4f78-b57b-ce8a9211bd02\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" Jul 08 19:20:03.884092 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.883952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbbv\" (UniqueName: \"kubernetes.io/projected/5636bf60-dc82-4f78-b57b-ce8a9211bd02-kube-api-access-9tbbv\") pod \"postgres-69fb4594cf-c8qq6\" (UID: \"5636bf60-dc82-4f78-b57b-ce8a9211bd02\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" Jul 08 19:20:03.985030 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.984998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbbv\" (UniqueName: \"kubernetes.io/projected/5636bf60-dc82-4f78-b57b-ce8a9211bd02-kube-api-access-9tbbv\") pod \"postgres-69fb4594cf-c8qq6\" (UID: \"5636bf60-dc82-4f78-b57b-ce8a9211bd02\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" Jul 08 19:20:03.985193 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.985076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5636bf60-dc82-4f78-b57b-ce8a9211bd02-data\") pod \"postgres-69fb4594cf-c8qq6\" (UID: \"5636bf60-dc82-4f78-b57b-ce8a9211bd02\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" Jul 08 19:20:03.985434 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.985419 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5636bf60-dc82-4f78-b57b-ce8a9211bd02-data\") pod \"postgres-69fb4594cf-c8qq6\" (UID: \"5636bf60-dc82-4f78-b57b-ce8a9211bd02\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" Jul 08 19:20:03.994590 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:03.994567 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbbv\" (UniqueName: \"kubernetes.io/projected/5636bf60-dc82-4f78-b57b-ce8a9211bd02-kube-api-access-9tbbv\") pod \"postgres-69fb4594cf-c8qq6\" (UID: \"5636bf60-dc82-4f78-b57b-ce8a9211bd02\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" Jul 08 19:20:04.113321 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:04.113238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" Jul 08 19:20:04.249727 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:04.249699 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6"] Jul 08 19:20:04.250998 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:20:04.250973 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5636bf60_dc82_4f78_b57b_ce8a9211bd02.slice/crio-b3c5bf4149f78a8e76e8330289ad12907aab6e3a01bcf96f7fbb1383875c2521 WatchSource:0}: Error finding container b3c5bf4149f78a8e76e8330289ad12907aab6e3a01bcf96f7fbb1383875c2521: Status 404 returned error can't find the container with id b3c5bf4149f78a8e76e8330289ad12907aab6e3a01bcf96f7fbb1383875c2521 Jul 08 19:20:04.724930 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:04.724901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" event={"ID":"5636bf60-dc82-4f78-b57b-ce8a9211bd02","Type":"ContainerStarted","Data":"b3c5bf4149f78a8e76e8330289ad12907aab6e3a01bcf96f7fbb1383875c2521"} Jul 08 19:20:09.755326 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:09.755289 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" event={"ID":"5636bf60-dc82-4f78-b57b-ce8a9211bd02","Type":"ContainerStarted","Data":"f0212968ac569d1a25a77851d58768f480f63f61f6be8fb1797e4f3b6074dc6a"} Jul 08 19:20:09.755767 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:09.755477 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" Jul 08 19:20:09.773693 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:09.773650 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" podStartSLOduration=2.089120024 podStartE2EDuration="6.773636386s" totalImagesPullingTime="4.684516362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:20:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:20:09.772767243 +0000 UTC m=+527.075071409" watchObservedRunningTime="2026-07-08 19:20:09.773636386 +0000 UTC m=+527.075940553" Jul 08 19:20:15.790488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:15.790412 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c8qq6" Jul 08 19:20:16.350551 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.350516 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2lzs9"] Jul 08 19:20:16.357150 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.357128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2lzs9" Jul 08 19:20:16.361122 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.361099 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2lzs9"] Jul 08 19:20:16.486454 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.486419 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rccwq\" (UniqueName: \"kubernetes.io/projected/3103c202-66ae-4a50-91fc-166a6107179e-kube-api-access-rccwq\") pod \"authorino-5d4b67c446-2lzs9\" (UID: \"3103c202-66ae-4a50-91fc-166a6107179e\") " pod="kuadrant-system/authorino-5d4b67c446-2lzs9" Jul 08 19:20:16.586887 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.586860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rccwq\" (UniqueName: \"kubernetes.io/projected/3103c202-66ae-4a50-91fc-166a6107179e-kube-api-access-rccwq\") pod \"authorino-5d4b67c446-2lzs9\" (UID: \"3103c202-66ae-4a50-91fc-166a6107179e\") " pod="kuadrant-system/authorino-5d4b67c446-2lzs9" Jul 08 19:20:16.587869 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.587844 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2lzs9"] Jul 08 19:20:16.588068 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:20:16.588050 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rccwq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-2lzs9" podUID="3103c202-66ae-4a50-91fc-166a6107179e" Jul 08 19:20:16.599129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.599099 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rccwq\" (UniqueName: \"kubernetes.io/projected/3103c202-66ae-4a50-91fc-166a6107179e-kube-api-access-rccwq\") pod \"authorino-5d4b67c446-2lzs9\" (UID: \"3103c202-66ae-4a50-91fc-166a6107179e\") " pod="kuadrant-system/authorino-5d4b67c446-2lzs9" Jul 08 19:20:16.611754 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.611686 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54d67c9bdc-d6z8k"] Jul 08 19:20:16.615835 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.615816 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" Jul 08 19:20:16.627259 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.627239 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54d67c9bdc-d6z8k"] Jul 08 19:20:16.687988 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.687960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wh92\" (UniqueName: \"kubernetes.io/projected/07a4ef36-8551-48cd-9f76-387ecc8ef3b1-kube-api-access-6wh92\") pod \"authorino-54d67c9bdc-d6z8k\" (UID: \"07a4ef36-8551-48cd-9f76-387ecc8ef3b1\") " pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" Jul 08 19:20:16.789213 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.789177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wh92\" (UniqueName: \"kubernetes.io/projected/07a4ef36-8551-48cd-9f76-387ecc8ef3b1-kube-api-access-6wh92\") pod \"authorino-54d67c9bdc-d6z8k\" (UID: \"07a4ef36-8551-48cd-9f76-387ecc8ef3b1\") " pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" Jul 08 19:20:16.790078 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.790059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2lzs9" Jul 08 19:20:16.797026 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.797000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2lzs9" Jul 08 19:20:16.798152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.798135 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wh92\" (UniqueName: \"kubernetes.io/projected/07a4ef36-8551-48cd-9f76-387ecc8ef3b1-kube-api-access-6wh92\") pod \"authorino-54d67c9bdc-d6z8k\" (UID: \"07a4ef36-8551-48cd-9f76-387ecc8ef3b1\") " pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" Jul 08 19:20:16.862478 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.862400 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54d67c9bdc-d6z8k"] Jul 08 19:20:16.862631 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.862618 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" Jul 08 19:20:16.885404 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.885369 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b44577cbc-8htlr"] Jul 08 19:20:16.888832 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.888814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b44577cbc-8htlr" Jul 08 19:20:16.890444 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.890412 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rccwq\" (UniqueName: \"kubernetes.io/projected/3103c202-66ae-4a50-91fc-166a6107179e-kube-api-access-rccwq\") pod \"3103c202-66ae-4a50-91fc-166a6107179e\" (UID: \"3103c202-66ae-4a50-91fc-166a6107179e\") " Jul 08 19:20:16.892194 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.892175 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 08 19:20:16.893058 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.893025 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3103c202-66ae-4a50-91fc-166a6107179e-kube-api-access-rccwq" (OuterVolumeSpecName: "kube-api-access-rccwq") pod "3103c202-66ae-4a50-91fc-166a6107179e" (UID: "3103c202-66ae-4a50-91fc-166a6107179e"). InnerVolumeSpecName "kube-api-access-rccwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:20:16.897906 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.897883 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b44577cbc-8htlr"] Jul 08 19:20:16.991598 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.991559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6nsn\" (UniqueName: \"kubernetes.io/projected/0a0757d9-4929-46a6-b70d-af9285bd12ad-kube-api-access-l6nsn\") pod \"authorino-5b44577cbc-8htlr\" (UID: \"0a0757d9-4929-46a6-b70d-af9285bd12ad\") " pod="kuadrant-system/authorino-5b44577cbc-8htlr" Jul 08 19:20:16.991755 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.991627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a0757d9-4929-46a6-b70d-af9285bd12ad-tls-cert\") pod \"authorino-5b44577cbc-8htlr\" (UID: \"0a0757d9-4929-46a6-b70d-af9285bd12ad\") " pod="kuadrant-system/authorino-5b44577cbc-8htlr" Jul 08 19:20:16.991755 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:16.991686 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rccwq\" (UniqueName: \"kubernetes.io/projected/3103c202-66ae-4a50-91fc-166a6107179e-kube-api-access-rccwq\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:20:17.003624 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.003602 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54d67c9bdc-d6z8k"] Jul 08 19:20:17.005473 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:20:17.005448 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07a4ef36_8551_48cd_9f76_387ecc8ef3b1.slice/crio-8ebdebe955103a8e3cd3b14322616774b698b67131b787741079a5c330c410e2 WatchSource:0}: Error finding container 8ebdebe955103a8e3cd3b14322616774b698b67131b787741079a5c330c410e2: Status 404 returned error can't find the container with id 8ebdebe955103a8e3cd3b14322616774b698b67131b787741079a5c330c410e2 Jul 08 19:20:17.092680 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.092644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6nsn\" (UniqueName: \"kubernetes.io/projected/0a0757d9-4929-46a6-b70d-af9285bd12ad-kube-api-access-l6nsn\") pod \"authorino-5b44577cbc-8htlr\" (UID: \"0a0757d9-4929-46a6-b70d-af9285bd12ad\") " pod="kuadrant-system/authorino-5b44577cbc-8htlr" Jul 08 19:20:17.092844 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.092694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a0757d9-4929-46a6-b70d-af9285bd12ad-tls-cert\") pod \"authorino-5b44577cbc-8htlr\" (UID: \"0a0757d9-4929-46a6-b70d-af9285bd12ad\") " pod="kuadrant-system/authorino-5b44577cbc-8htlr" Jul 08 19:20:17.095042 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.095016 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a0757d9-4929-46a6-b70d-af9285bd12ad-tls-cert\") pod \"authorino-5b44577cbc-8htlr\" (UID: \"0a0757d9-4929-46a6-b70d-af9285bd12ad\") " pod="kuadrant-system/authorino-5b44577cbc-8htlr" Jul 08 19:20:17.111912 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.111885 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6nsn\" (UniqueName: \"kubernetes.io/projected/0a0757d9-4929-46a6-b70d-af9285bd12ad-kube-api-access-l6nsn\") pod \"authorino-5b44577cbc-8htlr\" (UID: \"0a0757d9-4929-46a6-b70d-af9285bd12ad\") " pod="kuadrant-system/authorino-5b44577cbc-8htlr" Jul 08 19:20:17.205081 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.205003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b44577cbc-8htlr" Jul 08 19:20:17.337311 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.337291 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b44577cbc-8htlr"] Jul 08 19:20:17.338696 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:20:17.338671 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a0757d9_4929_46a6_b70d_af9285bd12ad.slice/crio-c8f857bf1b04044bd7217914840e3592bc0c4e694073da4abc8b1c9a53d484b3 WatchSource:0}: Error finding container c8f857bf1b04044bd7217914840e3592bc0c4e694073da4abc8b1c9a53d484b3: Status 404 returned error can't find the container with id c8f857bf1b04044bd7217914840e3592bc0c4e694073da4abc8b1c9a53d484b3 Jul 08 19:20:17.797815 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.797724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b44577cbc-8htlr" event={"ID":"0a0757d9-4929-46a6-b70d-af9285bd12ad","Type":"ContainerStarted","Data":"2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9"} Jul 08 19:20:17.797815 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.797779 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b44577cbc-8htlr" event={"ID":"0a0757d9-4929-46a6-b70d-af9285bd12ad","Type":"ContainerStarted","Data":"c8f857bf1b04044bd7217914840e3592bc0c4e694073da4abc8b1c9a53d484b3"} Jul 08 19:20:17.800395 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.800354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2lzs9" Jul 08 19:20:17.800545 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.800414 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" podUID="07a4ef36-8551-48cd-9f76-387ecc8ef3b1" containerName="authorino" containerID="cri-o://e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3" gracePeriod=30 Jul 08 19:20:17.800545 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.800336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" event={"ID":"07a4ef36-8551-48cd-9f76-387ecc8ef3b1","Type":"ContainerStarted","Data":"e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3"} Jul 08 19:20:17.800545 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.800460 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" event={"ID":"07a4ef36-8551-48cd-9f76-387ecc8ef3b1","Type":"ContainerStarted","Data":"8ebdebe955103a8e3cd3b14322616774b698b67131b787741079a5c330c410e2"} Jul 08 19:20:17.816857 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.816812 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b44577cbc-8htlr" podStartSLOduration=1.5091035449999999 podStartE2EDuration="1.816796484s" totalImagesPullingTime="307.692939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:20:17.81520426 +0000 UTC m=+535.117508428" watchObservedRunningTime="2026-07-08 19:20:17.816796484 +0000 UTC m=+535.119100650" Jul 08 19:20:17.849735 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.849698 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tth58"] Jul 08 19:20:17.849973 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.849868 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-tth58" podUID="3bf6fc20-64b7-43db-9132-3612c6036872" containerName="authorino" containerID="cri-o://1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544" gracePeriod=30 Jul 08 19:20:17.856367 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.856326 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2lzs9"] Jul 08 19:20:17.866313 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.866289 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2lzs9"] Jul 08 19:20:17.884682 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:17.884641 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" podStartSLOduration=1.393215437 podStartE2EDuration="1.884627271s" totalImagesPullingTime="491.411834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:20:17.88286298 +0000 UTC m=+535.185167149" watchObservedRunningTime="2026-07-08 19:20:17.884627271 +0000 UTC m=+535.186931437" Jul 08 19:20:18.109771 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.109750 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" Jul 08 19:20:18.111182 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.111165 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tth58" Jul 08 19:20:18.202978 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.202944 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggrc9\" (UniqueName: \"kubernetes.io/projected/3bf6fc20-64b7-43db-9132-3612c6036872-kube-api-access-ggrc9\") pod \"3bf6fc20-64b7-43db-9132-3612c6036872\" (UID: \"3bf6fc20-64b7-43db-9132-3612c6036872\") " Jul 08 19:20:18.203144 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.202987 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wh92\" (UniqueName: \"kubernetes.io/projected/07a4ef36-8551-48cd-9f76-387ecc8ef3b1-kube-api-access-6wh92\") pod \"07a4ef36-8551-48cd-9f76-387ecc8ef3b1\" (UID: \"07a4ef36-8551-48cd-9f76-387ecc8ef3b1\") " Jul 08 19:20:18.205096 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.205066 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf6fc20-64b7-43db-9132-3612c6036872-kube-api-access-ggrc9" (OuterVolumeSpecName: "kube-api-access-ggrc9") pod "3bf6fc20-64b7-43db-9132-3612c6036872" (UID: "3bf6fc20-64b7-43db-9132-3612c6036872"). InnerVolumeSpecName "kube-api-access-ggrc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:20:18.205209 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.205074 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a4ef36-8551-48cd-9f76-387ecc8ef3b1-kube-api-access-6wh92" (OuterVolumeSpecName: "kube-api-access-6wh92") pod "07a4ef36-8551-48cd-9f76-387ecc8ef3b1" (UID: "07a4ef36-8551-48cd-9f76-387ecc8ef3b1"). InnerVolumeSpecName "kube-api-access-6wh92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:20:18.304374 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.304308 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggrc9\" (UniqueName: \"kubernetes.io/projected/3bf6fc20-64b7-43db-9132-3612c6036872-kube-api-access-ggrc9\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:20:18.304374 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.304371 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wh92\" (UniqueName: \"kubernetes.io/projected/07a4ef36-8551-48cd-9f76-387ecc8ef3b1-kube-api-access-6wh92\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:20:18.806550 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.806518 2576 generic.go:356] "Generic (PLEG): container finished" podID="07a4ef36-8551-48cd-9f76-387ecc8ef3b1" containerID="e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3" exitCode=0 Jul 08 19:20:18.807013 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.806559 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" Jul 08 19:20:18.807013 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.806603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" event={"ID":"07a4ef36-8551-48cd-9f76-387ecc8ef3b1","Type":"ContainerDied","Data":"e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3"} Jul 08 19:20:18.807013 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.806646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54d67c9bdc-d6z8k" event={"ID":"07a4ef36-8551-48cd-9f76-387ecc8ef3b1","Type":"ContainerDied","Data":"8ebdebe955103a8e3cd3b14322616774b698b67131b787741079a5c330c410e2"} Jul 08 19:20:18.807013 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.806668 2576 scope.go:117] "RemoveContainer" containerID="e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3" Jul 08 19:20:18.807959 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.807937 2576 generic.go:356] "Generic (PLEG): container finished" podID="3bf6fc20-64b7-43db-9132-3612c6036872" containerID="1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544" exitCode=0 Jul 08 19:20:18.808090 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.807974 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-tth58" Jul 08 19:20:18.808090 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.808030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tth58" event={"ID":"3bf6fc20-64b7-43db-9132-3612c6036872","Type":"ContainerDied","Data":"1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544"} Jul 08 19:20:18.808090 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.808068 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-tth58" event={"ID":"3bf6fc20-64b7-43db-9132-3612c6036872","Type":"ContainerDied","Data":"6895aed4109c28fe3ba0141c43abeca008a26b93fc7b366e5eac1bc2871745b8"} Jul 08 19:20:18.816334 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.816307 2576 scope.go:117] "RemoveContainer" containerID="e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3" Jul 08 19:20:18.816613 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:20:18.816587 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3\": container with ID starting with e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3 not found: ID does not exist" containerID="e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3" Jul 08 19:20:18.816721 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.816619 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3"} err="failed to get container status \"e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3\": rpc error: code = NotFound desc = could not find container \"e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3\": container with ID starting with e30883b066317851cb8c2d277f352b74f3512d972bc42b2f2b81f008ed9470c3 not found: ID does not exist" Jul 08 19:20:18.816721 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.816634 2576 scope.go:117] "RemoveContainer" containerID="1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544" Jul 08 19:20:18.825082 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.825068 2576 scope.go:117] "RemoveContainer" containerID="1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544" Jul 08 19:20:18.825320 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:20:18.825305 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544\": container with ID starting with 1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544 not found: ID does not exist" containerID="1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544" Jul 08 19:20:18.825387 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.825326 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544"} err="failed to get container status \"1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544\": rpc error: code = NotFound desc = could not find container \"1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544\": container with ID starting with 1f58bd0a73970c1a2c7dce154d811100df26be642bff09390546c13fb9923544 not found: ID does not exist" Jul 08 19:20:18.834131 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.834110 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54d67c9bdc-d6z8k"] Jul 08 19:20:18.840580 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.840562 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54d67c9bdc-d6z8k"] Jul 08 19:20:18.851992 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.851967 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tth58"] Jul 08 19:20:18.856777 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:18.856724 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-tth58"] Jul 08 19:20:19.267687 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:19.267608 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07a4ef36-8551-48cd-9f76-387ecc8ef3b1" path="/var/lib/kubelet/pods/07a4ef36-8551-48cd-9f76-387ecc8ef3b1/volumes" Jul 08 19:20:19.267935 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:19.267923 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3103c202-66ae-4a50-91fc-166a6107179e" path="/var/lib/kubelet/pods/3103c202-66ae-4a50-91fc-166a6107179e/volumes" Jul 08 19:20:19.268112 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:20:19.268101 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf6fc20-64b7-43db-9132-3612c6036872" path="/var/lib/kubelet/pods/3bf6fc20-64b7-43db-9132-3612c6036872/volumes" Jul 08 19:21:17.403009 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.402974 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn"] Jul 08 19:21:17.403560 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.403529 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf6fc20-64b7-43db-9132-3612c6036872" containerName="authorino" Jul 08 19:21:17.403560 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.403549 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf6fc20-64b7-43db-9132-3612c6036872" containerName="authorino" Jul 08 19:21:17.403710 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.403564 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a4ef36-8551-48cd-9f76-387ecc8ef3b1" containerName="authorino" Jul 08 19:21:17.403710 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.403572 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a4ef36-8551-48cd-9f76-387ecc8ef3b1" containerName="authorino" Jul 08 19:21:17.403710 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.403663 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="07a4ef36-8551-48cd-9f76-387ecc8ef3b1" containerName="authorino" Jul 08 19:21:17.403710 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.403676 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf6fc20-64b7-43db-9132-3612c6036872" containerName="authorino" Jul 08 19:21:17.406189 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.406167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.410577 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.410553 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h65ml\"" Jul 08 19:21:17.410699 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.410592 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 19:21:17.412316 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.412296 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 08 19:21:17.412454 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.412319 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 19:21:17.428327 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.428306 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn"] Jul 08 19:21:17.501687 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.501655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.501865 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.501704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.501865 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.501806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpjg\" (UniqueName: \"kubernetes.io/projected/24bbd019-1168-4202-9c54-e441d958c545-kube-api-access-jwpjg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.501978 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.501863 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.501978 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.501924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24bbd019-1168-4202-9c54-e441d958c545-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.502063 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.501999 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.603167 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.603134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24bbd019-1168-4202-9c54-e441d958c545-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.603380 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.603187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.603380 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.603209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.603380 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.603232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.603380 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.603257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpjg\" (UniqueName: \"kubernetes.io/projected/24bbd019-1168-4202-9c54-e441d958c545-kube-api-access-jwpjg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.603380 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.603303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.603687 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.603592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.603736 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.603693 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.603736 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.603728 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.605519 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.605499 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24bbd019-1168-4202-9c54-e441d958c545-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.605835 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.605817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24bbd019-1168-4202-9c54-e441d958c545-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.614717 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.614688 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpjg\" (UniqueName: \"kubernetes.io/projected/24bbd019-1168-4202-9c54-e441d958c545-kube-api-access-jwpjg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn\" (UID: \"24bbd019-1168-4202-9c54-e441d958c545\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.716890 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.716807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:17.859111 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:17.859072 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn"] Jul 08 19:21:17.862001 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:21:17.861965 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bbd019_1168_4202_9c54_e441d958c545.slice/crio-63010d664f345317c9adb9eb3d8043be2c3bb1708ea1e520b3ae781000c77ce8 WatchSource:0}: Error finding container 63010d664f345317c9adb9eb3d8043be2c3bb1708ea1e520b3ae781000c77ce8: Status 404 returned error can't find the container with id 63010d664f345317c9adb9eb3d8043be2c3bb1708ea1e520b3ae781000c77ce8 Jul 08 19:21:18.104900 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:18.104860 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" event={"ID":"24bbd019-1168-4202-9c54-e441d958c545","Type":"ContainerStarted","Data":"63010d664f345317c9adb9eb3d8043be2c3bb1708ea1e520b3ae781000c77ce8"} Jul 08 19:21:20.118829 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:20.118794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" event={"ID":"24bbd019-1168-4202-9c54-e441d958c545","Type":"ContainerStarted","Data":"5a5bc038cd19b1655a07e932d95591550325dc6c809fe48acc9a7b26f00891d3"} Jul 08 19:21:23.137945 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:23.137914 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:21:23.138418 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:23.138035 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:21:27.717567 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:27.717519 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:27.719924 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:27.717599 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:27.734441 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:27.734417 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:27.753189 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:27.753146 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" podStartSLOduration=9.486627686 podStartE2EDuration="10.753130559s" totalImagesPullingTime="1.266502873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:21:20.151630849 +0000 UTC m=+597.453935017" watchObservedRunningTime="2026-07-08 19:21:27.753130559 +0000 UTC m=+605.055434726" Jul 08 19:21:28.171167 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:28.171140 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn" Jul 08 19:21:52.687039 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.687005 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm"] Jul 08 19:21:52.692221 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.692196 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.695729 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.695707 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 08 19:21:52.704029 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.704008 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm"] Jul 08 19:21:52.783496 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.783458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.783496 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.783494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.783712 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.783523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qzr\" (UniqueName: \"kubernetes.io/projected/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-kube-api-access-49qzr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.783712 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.783541 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.783712 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.783607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.783712 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.783649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.884821 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.884774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qzr\" (UniqueName: \"kubernetes.io/projected/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-kube-api-access-49qzr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.884821 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.884825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.885049 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.884868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.885049 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.884900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.885049 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.884963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.885049 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.884988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.885410 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.885386 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.885534 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.885392 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.885534 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.885473 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.887234 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.887213 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.887334 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.887305 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:52.893420 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:52.893400 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qzr\" (UniqueName: \"kubernetes.io/projected/5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf-kube-api-access-49qzr\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm\" (UID: \"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:53.002908 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:53.002815 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:21:53.140304 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:53.140276 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm"] Jul 08 19:21:53.141377 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:21:53.141354 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7eb4f8_b60b_42ab_a1c6_d5a1bc4c4dbf.slice/crio-44414d9cc513226ac3190b993875e9a64989d718c2334a844428418a394020cf WatchSource:0}: Error finding container 44414d9cc513226ac3190b993875e9a64989d718c2334a844428418a394020cf: Status 404 returned error can't find the container with id 44414d9cc513226ac3190b993875e9a64989d718c2334a844428418a394020cf Jul 08 19:21:53.142706 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:53.142687 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:21:53.287339 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:53.287311 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" event={"ID":"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf","Type":"ContainerStarted","Data":"44414d9cc513226ac3190b993875e9a64989d718c2334a844428418a394020cf"} Jul 08 19:21:54.295488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:54.295449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" event={"ID":"5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf","Type":"ContainerStarted","Data":"c379f277d948e7bf48fcc207855aec84e0619dd74a5cbcc249ffbc6118e73e88"} Jul 08 19:21:54.316678 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:21:54.316599 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" podStartSLOduration=2.130590648 podStartE2EDuration="2.316583184s" totalImagesPullingTime="185.992536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:21:54.31552391 +0000 UTC m=+631.617828100" watchObservedRunningTime="2026-07-08 19:21:54.316583184 +0000 UTC m=+631.618887352" Jul 08 19:22:03.003815 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:22:03.003756 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:22:03.004271 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:22:03.003859 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:22:03.016425 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:22:03.016397 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:22:03.350897 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:22:03.350865 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm" Jul 08 19:23:25.688883 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:25.688810 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65dc567574-m7dhm"] Jul 08 19:23:25.692459 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:25.692439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65dc567574-m7dhm" Jul 08 19:23:25.701053 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:25.701032 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65dc567574-m7dhm"] Jul 08 19:23:25.772029 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:25.771998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d0ff258c-cca7-4501-8432-e7c7318607b2-tls-cert\") pod \"authorino-65dc567574-m7dhm\" (UID: \"d0ff258c-cca7-4501-8432-e7c7318607b2\") " pod="kuadrant-system/authorino-65dc567574-m7dhm" Jul 08 19:23:25.772208 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:25.772041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz42z\" (UniqueName: \"kubernetes.io/projected/d0ff258c-cca7-4501-8432-e7c7318607b2-kube-api-access-sz42z\") pod \"authorino-65dc567574-m7dhm\" (UID: \"d0ff258c-cca7-4501-8432-e7c7318607b2\") " pod="kuadrant-system/authorino-65dc567574-m7dhm" Jul 08 19:23:25.873267 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:25.873230 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d0ff258c-cca7-4501-8432-e7c7318607b2-tls-cert\") pod \"authorino-65dc567574-m7dhm\" (UID: \"d0ff258c-cca7-4501-8432-e7c7318607b2\") " pod="kuadrant-system/authorino-65dc567574-m7dhm" Jul 08 19:23:25.873460 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:25.873282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz42z\" (UniqueName: \"kubernetes.io/projected/d0ff258c-cca7-4501-8432-e7c7318607b2-kube-api-access-sz42z\") pod \"authorino-65dc567574-m7dhm\" (UID: \"d0ff258c-cca7-4501-8432-e7c7318607b2\") " pod="kuadrant-system/authorino-65dc567574-m7dhm" Jul 08 19:23:25.875810 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:25.875778 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d0ff258c-cca7-4501-8432-e7c7318607b2-tls-cert\") pod \"authorino-65dc567574-m7dhm\" (UID: \"d0ff258c-cca7-4501-8432-e7c7318607b2\") " pod="kuadrant-system/authorino-65dc567574-m7dhm" Jul 08 19:23:25.884176 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:25.884144 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz42z\" (UniqueName: \"kubernetes.io/projected/d0ff258c-cca7-4501-8432-e7c7318607b2-kube-api-access-sz42z\") pod \"authorino-65dc567574-m7dhm\" (UID: \"d0ff258c-cca7-4501-8432-e7c7318607b2\") " pod="kuadrant-system/authorino-65dc567574-m7dhm" Jul 08 19:23:26.002643 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:26.002562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65dc567574-m7dhm" Jul 08 19:23:26.151368 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:26.151327 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65dc567574-m7dhm"] Jul 08 19:23:26.153296 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:23:26.153264 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0ff258c_cca7_4501_8432_e7c7318607b2.slice/crio-16f8f28d9e6defb5ffcc8c156fb172fc73c00260f458bae003173410c2e05fbc WatchSource:0}: Error finding container 16f8f28d9e6defb5ffcc8c156fb172fc73c00260f458bae003173410c2e05fbc: Status 404 returned error can't find the container with id 16f8f28d9e6defb5ffcc8c156fb172fc73c00260f458bae003173410c2e05fbc Jul 08 19:23:26.783038 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:26.783000 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65dc567574-m7dhm" event={"ID":"d0ff258c-cca7-4501-8432-e7c7318607b2","Type":"ContainerStarted","Data":"b3e78d32210e09a4bf40d70f9b7ff2b609286f4a5f0095e2cd6ad9e5728f0c22"} Jul 08 19:23:26.783486 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:26.783049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65dc567574-m7dhm" event={"ID":"d0ff258c-cca7-4501-8432-e7c7318607b2","Type":"ContainerStarted","Data":"16f8f28d9e6defb5ffcc8c156fb172fc73c00260f458bae003173410c2e05fbc"} Jul 08 19:23:26.806088 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:26.806030 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65dc567574-m7dhm" podStartSLOduration=1.388540033 podStartE2EDuration="1.806007633s" totalImagesPullingTime="417.4676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:23:26.805218285 +0000 UTC m=+724.107522453" watchObservedRunningTime="2026-07-08 19:23:26.806007633 +0000 UTC m=+724.108311801" Jul 08 19:23:26.844420 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:26.843824 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b44577cbc-8htlr"] Jul 08 19:23:26.844420 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:26.844054 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5b44577cbc-8htlr" podUID="0a0757d9-4929-46a6-b70d-af9285bd12ad" containerName="authorino" containerID="cri-o://2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9" gracePeriod=30 Jul 08 19:23:27.111479 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.111452 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b44577cbc-8htlr" Jul 08 19:23:27.284414 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.284384 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6nsn\" (UniqueName: \"kubernetes.io/projected/0a0757d9-4929-46a6-b70d-af9285bd12ad-kube-api-access-l6nsn\") pod \"0a0757d9-4929-46a6-b70d-af9285bd12ad\" (UID: \"0a0757d9-4929-46a6-b70d-af9285bd12ad\") " Jul 08 19:23:27.284562 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.284470 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a0757d9-4929-46a6-b70d-af9285bd12ad-tls-cert\") pod \"0a0757d9-4929-46a6-b70d-af9285bd12ad\" (UID: \"0a0757d9-4929-46a6-b70d-af9285bd12ad\") " Jul 08 19:23:27.286602 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.286572 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0757d9-4929-46a6-b70d-af9285bd12ad-kube-api-access-l6nsn" (OuterVolumeSpecName: "kube-api-access-l6nsn") pod "0a0757d9-4929-46a6-b70d-af9285bd12ad" (UID: "0a0757d9-4929-46a6-b70d-af9285bd12ad"). InnerVolumeSpecName "kube-api-access-l6nsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:23:27.295745 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.295720 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0757d9-4929-46a6-b70d-af9285bd12ad-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0a0757d9-4929-46a6-b70d-af9285bd12ad" (UID: "0a0757d9-4929-46a6-b70d-af9285bd12ad"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:23:27.385850 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.385770 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a0757d9-4929-46a6-b70d-af9285bd12ad-tls-cert\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:23:27.385850 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.385794 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6nsn\" (UniqueName: \"kubernetes.io/projected/0a0757d9-4929-46a6-b70d-af9285bd12ad-kube-api-access-l6nsn\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:23:27.790185 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.790149 2576 generic.go:356] "Generic (PLEG): container finished" podID="0a0757d9-4929-46a6-b70d-af9285bd12ad" containerID="2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9" exitCode=0 Jul 08 19:23:27.790614 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.790196 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b44577cbc-8htlr" Jul 08 19:23:27.790614 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.790236 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b44577cbc-8htlr" event={"ID":"0a0757d9-4929-46a6-b70d-af9285bd12ad","Type":"ContainerDied","Data":"2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9"} Jul 08 19:23:27.790614 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.790289 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b44577cbc-8htlr" event={"ID":"0a0757d9-4929-46a6-b70d-af9285bd12ad","Type":"ContainerDied","Data":"c8f857bf1b04044bd7217914840e3592bc0c4e694073da4abc8b1c9a53d484b3"} Jul 08 19:23:27.790614 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.790313 2576 scope.go:117] "RemoveContainer" containerID="2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9" Jul 08 19:23:27.801373 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.801331 2576 scope.go:117] "RemoveContainer" containerID="2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9" Jul 08 19:23:27.801664 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:23:27.801644 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9\": container with ID starting with 2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9 not found: ID does not exist" containerID="2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9" Jul 08 19:23:27.801729 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.801676 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9"} err="failed to get container status \"2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9\": rpc error: code = NotFound desc = could not find container \"2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9\": container with ID starting with 2ddcfe2df4432816cd693c682361e951da9a12d35026dc0565be5780e02135b9 not found: ID does not exist" Jul 08 19:23:27.819775 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.819747 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b44577cbc-8htlr"] Jul 08 19:23:27.826026 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:27.825996 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b44577cbc-8htlr"] Jul 08 19:23:29.267987 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:23:29.267953 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a0757d9-4929-46a6-b70d-af9285bd12ad" path="/var/lib/kubelet/pods/0a0757d9-4929-46a6-b70d-af9285bd12ad/volumes" Jul 08 19:26:23.173236 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:26:23.173169 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:26:23.174692 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:26:23.174667 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:29:33.926816 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:33.926720 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7"] Jul 08 19:29:33.929394 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:33.926930 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" podUID="65b32cfb-564f-4e7a-981a-f724d35c1716" containerName="manager" containerID="cri-o://62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a" gracePeriod=10 Jul 08 19:29:36.488369 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.488332 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:29:36.564300 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.564272 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65b32cfb-564f-4e7a-981a-f724d35c1716-extensions-socket-volume\") pod \"65b32cfb-564f-4e7a-981a-f724d35c1716\" (UID: \"65b32cfb-564f-4e7a-981a-f724d35c1716\") " Jul 08 19:29:36.564476 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.564310 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swtxf\" (UniqueName: \"kubernetes.io/projected/65b32cfb-564f-4e7a-981a-f724d35c1716-kube-api-access-swtxf\") pod \"65b32cfb-564f-4e7a-981a-f724d35c1716\" (UID: \"65b32cfb-564f-4e7a-981a-f724d35c1716\") " Jul 08 19:29:36.564659 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.564637 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b32cfb-564f-4e7a-981a-f724d35c1716-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "65b32cfb-564f-4e7a-981a-f724d35c1716" (UID: "65b32cfb-564f-4e7a-981a-f724d35c1716"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:29:36.566373 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.566338 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b32cfb-564f-4e7a-981a-f724d35c1716-kube-api-access-swtxf" (OuterVolumeSpecName: "kube-api-access-swtxf") pod "65b32cfb-564f-4e7a-981a-f724d35c1716" (UID: "65b32cfb-564f-4e7a-981a-f724d35c1716"). InnerVolumeSpecName "kube-api-access-swtxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:29:36.665166 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.665089 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65b32cfb-564f-4e7a-981a-f724d35c1716-extensions-socket-volume\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:29:36.665166 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.665116 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swtxf\" (UniqueName: \"kubernetes.io/projected/65b32cfb-564f-4e7a-981a-f724d35c1716-kube-api-access-swtxf\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:29:36.702624 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.702591 2576 generic.go:356] "Generic (PLEG): container finished" podID="65b32cfb-564f-4e7a-981a-f724d35c1716" containerID="62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a" exitCode=0 Jul 08 19:29:36.702731 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.702641 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" Jul 08 19:29:36.702731 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.702672 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" event={"ID":"65b32cfb-564f-4e7a-981a-f724d35c1716","Type":"ContainerDied","Data":"62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a"} Jul 08 19:29:36.702731 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.702723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7" event={"ID":"65b32cfb-564f-4e7a-981a-f724d35c1716","Type":"ContainerDied","Data":"e34504d74db8e2498caeb52a4e3179d3231b745793aac6e406b60802b625a7cd"} Jul 08 19:29:36.702884 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.702746 2576 scope.go:117] "RemoveContainer" containerID="62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a" Jul 08 19:29:36.719246 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.719228 2576 scope.go:117] "RemoveContainer" containerID="62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a" Jul 08 19:29:36.719578 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:29:36.719558 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a\": container with ID starting with 62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a not found: ID does not exist" containerID="62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a" Jul 08 19:29:36.719646 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.719585 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a"} err="failed to get container status \"62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a\": rpc error: code = NotFound desc = could not find container \"62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a\": container with ID starting with 62c8271f55dbea3165870288815edde482ed05f10a0d4067ae106084629de46a not found: ID does not exist" Jul 08 19:29:36.731714 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.731693 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7"] Jul 08 19:29:36.738390 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:36.738371 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z62d7"] Jul 08 19:29:37.268484 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:29:37.268455 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b32cfb-564f-4e7a-981a-f724d35c1716" path="/var/lib/kubelet/pods/65b32cfb-564f-4e7a-981a-f724d35c1716/volumes" Jul 08 19:30:00.150082 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.150046 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6"] Jul 08 19:30:00.150488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.150398 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a0757d9-4929-46a6-b70d-af9285bd12ad" containerName="authorino" Jul 08 19:30:00.150488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.150409 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a0757d9-4929-46a6-b70d-af9285bd12ad" containerName="authorino" Jul 08 19:30:00.150488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.150429 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b32cfb-564f-4e7a-981a-f724d35c1716" containerName="manager" Jul 08 19:30:00.150488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.150434 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b32cfb-564f-4e7a-981a-f724d35c1716" containerName="manager" Jul 08 19:30:00.150488 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.150487 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b32cfb-564f-4e7a-981a-f724d35c1716" containerName="manager" Jul 08 19:30:00.150654 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.150496 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a0757d9-4929-46a6-b70d-af9285bd12ad" containerName="authorino" Jul 08 19:30:00.153465 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.153449 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" Jul 08 19:30:00.157072 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.157051 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-58xsg\"" Jul 08 19:30:00.161829 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.161806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6"] Jul 08 19:30:00.266655 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.266624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54kg\" (UniqueName: \"kubernetes.io/projected/5e84d0e9-f589-489e-842e-cdaa9b2bff32-kube-api-access-z54kg\") pod \"maas-api-key-cleanup-29725650-vf7w6\" (UID: \"5e84d0e9-f589-489e-842e-cdaa9b2bff32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" Jul 08 19:30:00.367950 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.367915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54kg\" (UniqueName: \"kubernetes.io/projected/5e84d0e9-f589-489e-842e-cdaa9b2bff32-kube-api-access-z54kg\") pod \"maas-api-key-cleanup-29725650-vf7w6\" (UID: \"5e84d0e9-f589-489e-842e-cdaa9b2bff32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" Jul 08 19:30:00.379245 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.379217 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54kg\" (UniqueName: \"kubernetes.io/projected/5e84d0e9-f589-489e-842e-cdaa9b2bff32-kube-api-access-z54kg\") pod \"maas-api-key-cleanup-29725650-vf7w6\" (UID: \"5e84d0e9-f589-489e-842e-cdaa9b2bff32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" Jul 08 19:30:00.464599 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.464517 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" Jul 08 19:30:00.596626 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.596602 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6"] Jul 08 19:30:00.598186 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:30:00.598155 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e84d0e9_f589_489e_842e_cdaa9b2bff32.slice/crio-1d9a3dd6ec1692ecfee698ee494e138d35d36beed64c41ebecab1e5ff7deb42a WatchSource:0}: Error finding container 1d9a3dd6ec1692ecfee698ee494e138d35d36beed64c41ebecab1e5ff7deb42a: Status 404 returned error can't find the container with id 1d9a3dd6ec1692ecfee698ee494e138d35d36beed64c41ebecab1e5ff7deb42a Jul 08 19:30:00.600127 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.600104 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:30:00.829825 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:00.829789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" event={"ID":"5e84d0e9-f589-489e-842e-cdaa9b2bff32","Type":"ContainerStarted","Data":"1d9a3dd6ec1692ecfee698ee494e138d35d36beed64c41ebecab1e5ff7deb42a"} Jul 08 19:30:03.847223 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:03.847177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" event={"ID":"5e84d0e9-f589-489e-842e-cdaa9b2bff32","Type":"ContainerStarted","Data":"8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442"} Jul 08 19:30:03.864666 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:03.864617 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" podStartSLOduration=1.769555335 podStartE2EDuration="3.864602897s" totalImagesPullingTime="2.095047562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:30:03.863808601 +0000 UTC m=+1121.166112780" watchObservedRunningTime="2026-07-08 19:30:03.864602897 +0000 UTC m=+1121.166907063" Jul 08 19:30:35.102620 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.102586 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2"] Jul 08 19:30:35.107464 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.107440 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" Jul 08 19:30:35.111405 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.111388 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ztr8n\"" Jul 08 19:30:35.123735 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.123698 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2"] Jul 08 19:30:35.237744 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.237708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c942m\" (UniqueName: \"kubernetes.io/projected/cde2715b-b16b-4df2-87bc-c6d94b83736f-kube-api-access-c942m\") pod \"kuadrant-operator-controller-manager-86878c769f-47rb2\" (UID: \"cde2715b-b16b-4df2-87bc-c6d94b83736f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" Jul 08 19:30:35.237744 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.237749 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cde2715b-b16b-4df2-87bc-c6d94b83736f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-47rb2\" (UID: \"cde2715b-b16b-4df2-87bc-c6d94b83736f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" Jul 08 19:30:35.338321 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.338278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c942m\" (UniqueName: \"kubernetes.io/projected/cde2715b-b16b-4df2-87bc-c6d94b83736f-kube-api-access-c942m\") pod \"kuadrant-operator-controller-manager-86878c769f-47rb2\" (UID: \"cde2715b-b16b-4df2-87bc-c6d94b83736f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" Jul 08 19:30:35.338531 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.338458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cde2715b-b16b-4df2-87bc-c6d94b83736f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-47rb2\" (UID: \"cde2715b-b16b-4df2-87bc-c6d94b83736f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" Jul 08 19:30:35.338810 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.338788 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cde2715b-b16b-4df2-87bc-c6d94b83736f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-47rb2\" (UID: \"cde2715b-b16b-4df2-87bc-c6d94b83736f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" Jul 08 19:30:35.350739 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.350719 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c942m\" (UniqueName: \"kubernetes.io/projected/cde2715b-b16b-4df2-87bc-c6d94b83736f-kube-api-access-c942m\") pod \"kuadrant-operator-controller-manager-86878c769f-47rb2\" (UID: \"cde2715b-b16b-4df2-87bc-c6d94b83736f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" Jul 08 19:30:35.422504 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.422413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" Jul 08 19:30:35.564767 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:35.564736 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2"] Jul 08 19:30:35.568979 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:30:35.568946 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde2715b_b16b_4df2_87bc_c6d94b83736f.slice/crio-1d52229b99cde4c8ae3456ed95530f33214d9a41bea7ffb9a8cc550065e2e953 WatchSource:0}: Error finding container 1d52229b99cde4c8ae3456ed95530f33214d9a41bea7ffb9a8cc550065e2e953: Status 404 returned error can't find the container with id 1d52229b99cde4c8ae3456ed95530f33214d9a41bea7ffb9a8cc550065e2e953 Jul 08 19:30:36.012677 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:36.012640 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" event={"ID":"cde2715b-b16b-4df2-87bc-c6d94b83736f","Type":"ContainerStarted","Data":"58cb69d55b17beae32e7f20fb6d912d4eb2891c30bc4b94e819bee2f178e8f88"} Jul 08 19:30:36.012881 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:36.012700 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" event={"ID":"cde2715b-b16b-4df2-87bc-c6d94b83736f","Type":"ContainerStarted","Data":"1d52229b99cde4c8ae3456ed95530f33214d9a41bea7ffb9a8cc550065e2e953"} Jul 08 19:30:36.012881 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:36.012808 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" Jul 08 19:30:36.038291 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:36.038242 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" podStartSLOduration=1.038223863 podStartE2EDuration="1.038223863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:30:36.03567375 +0000 UTC m=+1153.337977918" watchObservedRunningTime="2026-07-08 19:30:36.038223863 +0000 UTC m=+1153.340528035" Jul 08 19:30:47.020366 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:30:47.020302 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47rb2" Jul 08 19:31:00.011673 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:00.011633 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6"] Jul 08 19:31:00.012117 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:00.011859 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" podUID="5e84d0e9-f589-489e-842e-cdaa9b2bff32" containerName="cleanup" containerID="cri-o://8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442" gracePeriod=30 Jul 08 19:31:23.208305 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:23.208278 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:31:23.210606 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:23.210586 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:31:30.205200 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.205175 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" Jul 08 19:31:30.304122 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.304096 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z54kg\" (UniqueName: \"kubernetes.io/projected/5e84d0e9-f589-489e-842e-cdaa9b2bff32-kube-api-access-z54kg\") pod \"5e84d0e9-f589-489e-842e-cdaa9b2bff32\" (UID: \"5e84d0e9-f589-489e-842e-cdaa9b2bff32\") " Jul 08 19:31:30.305408 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.305379 2576 generic.go:356] "Generic (PLEG): container finished" podID="5e84d0e9-f589-489e-842e-cdaa9b2bff32" containerID="8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442" exitCode=137 Jul 08 19:31:30.305537 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.305431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" event={"ID":"5e84d0e9-f589-489e-842e-cdaa9b2bff32","Type":"ContainerDied","Data":"8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442"} Jul 08 19:31:30.305537 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.305467 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" Jul 08 19:31:30.305537 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.305479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6" event={"ID":"5e84d0e9-f589-489e-842e-cdaa9b2bff32","Type":"ContainerDied","Data":"1d9a3dd6ec1692ecfee698ee494e138d35d36beed64c41ebecab1e5ff7deb42a"} Jul 08 19:31:30.305537 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.305501 2576 scope.go:117] "RemoveContainer" containerID="8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442" Jul 08 19:31:30.306557 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.306536 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e84d0e9-f589-489e-842e-cdaa9b2bff32-kube-api-access-z54kg" (OuterVolumeSpecName: "kube-api-access-z54kg") pod "5e84d0e9-f589-489e-842e-cdaa9b2bff32" (UID: "5e84d0e9-f589-489e-842e-cdaa9b2bff32"). InnerVolumeSpecName "kube-api-access-z54kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:31:30.317356 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.317325 2576 scope.go:117] "RemoveContainer" containerID="8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442" Jul 08 19:31:30.317654 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:31:30.317637 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442\": container with ID starting with 8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442 not found: ID does not exist" containerID="8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442" Jul 08 19:31:30.317702 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.317666 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442"} err="failed to get container status \"8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442\": rpc error: code = NotFound desc = could not find container \"8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442\": container with ID starting with 8e544dc128a717255a672e89c1c19c25347b74560ace37196482df97c9aad442 not found: ID does not exist" Jul 08 19:31:30.405753 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.405717 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54kg\" (UniqueName: \"kubernetes.io/projected/5e84d0e9-f589-489e-842e-cdaa9b2bff32-kube-api-access-z54kg\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:31:30.628695 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.628663 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6"] Jul 08 19:31:30.634973 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:30.634947 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725650-vf7w6"] Jul 08 19:31:31.271206 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:31:31.271166 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e84d0e9-f589-489e-842e-cdaa9b2bff32" path="/var/lib/kubelet/pods/5e84d0e9-f589-489e-842e-cdaa9b2bff32/volumes" Jul 08 19:36:23.242966 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:36:23.242935 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:36:23.245266 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:36:23.245246 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:41:23.276680 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:41:23.276583 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:41:23.282643 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:41:23.280562 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:43:10.382446 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.382400 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9"] Jul 08 19:43:10.383039 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.382980 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e84d0e9-f589-489e-842e-cdaa9b2bff32" containerName="cleanup" Jul 08 19:43:10.383039 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.382996 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e84d0e9-f589-489e-842e-cdaa9b2bff32" containerName="cleanup" Jul 08 19:43:10.383150 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.383074 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e84d0e9-f589-489e-842e-cdaa9b2bff32" containerName="cleanup" Jul 08 19:43:10.386279 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.386258 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.389152 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.389116 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-cf47b599-openshift-default-dockercfg-6vwmj\"" Jul 08 19:43:10.404147 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.404119 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9"] Jul 08 19:43:10.529322 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.529279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-socket\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.529322 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.529321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-podinfo\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.529591 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.529416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-data\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.529591 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.529490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47whp\" (UniqueName: \"kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-kube-api-access-47whp\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.529591 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.529529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-envoy\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.529591 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.529560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c91fd0-b7e6-42b8-83a8-30118600715e-istiod-ca-cert\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.529591 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.529586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-token\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.529867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.529610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-certs\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.529867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.529639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-credential-socket\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.630872 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.630828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c91fd0-b7e6-42b8-83a8-30118600715e-istiod-ca-cert\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.630872 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.630869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-token\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.630890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-certs\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.630916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-credential-socket\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.630955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-socket\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.630985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-podinfo\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631129 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.631062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-data\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631405 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.631241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47whp\" (UniqueName: \"kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-kube-api-access-47whp\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631405 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.631364 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-certs\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.631419 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-socket\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.631434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-envoy\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631485 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.631467 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-credential-socket\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631673 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.631651 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c91fd0-b7e6-42b8-83a8-30118600715e-istiod-ca-cert\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.631740 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.631710 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-data\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.633692 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.633630 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-envoy\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.633692 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.633644 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-podinfo\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.639865 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.639843 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47whp\" (UniqueName: \"kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-kube-api-access-47whp\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.639951 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.639920 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-token\") pod \"e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.696635 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.696599 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:10.842060 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.842036 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9"] Jul 08 19:43:10.843473 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:43:10.843429 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c91fd0_b7e6_42b8_83a8_30118600715e.slice/crio-99257bd385166971511f870b43965fa735a5c3abb072a451b951c356856c7579 WatchSource:0}: Error finding container 99257bd385166971511f870b43965fa735a5c3abb072a451b951c356856c7579: Status 404 returned error can't find the container with id 99257bd385166971511f870b43965fa735a5c3abb072a451b951c356856c7579 Jul 08 19:43:10.845717 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.845684 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:43:10.845800 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.845749 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:43:10.845800 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.845776 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:43:10.920951 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.920916 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" event={"ID":"80c91fd0-b7e6-42b8-83a8-30118600715e","Type":"ContainerStarted","Data":"0b54eb70bcb0db8561bcf1250d2abfaf1dbe4e80b8c50b091b48d6590f171a03"} Jul 08 19:43:10.921115 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.920977 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" event={"ID":"80c91fd0-b7e6-42b8-83a8-30118600715e","Type":"ContainerStarted","Data":"99257bd385166971511f870b43965fa735a5c3abb072a451b951c356856c7579"} Jul 08 19:43:10.941473 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:10.941416 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" podStartSLOduration=0.941395658 podStartE2EDuration="941.395658ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:43:10.940383346 +0000 UTC m=+1908.242687514" watchObservedRunningTime="2026-07-08 19:43:10.941395658 +0000 UTC m=+1908.243699826" Jul 08 19:43:11.697762 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:11.697732 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:11.702371 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:11.702329 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:11.926391 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:11.926358 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:11.927442 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:11.927422 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:16.838769 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:16.838731 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9"] Jul 08 19:43:16.839219 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:16.838980 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" podUID="80c91fd0-b7e6-42b8-83a8-30118600715e" containerName="istio-proxy" containerID="cri-o://0b54eb70bcb0db8561bcf1250d2abfaf1dbe4e80b8c50b091b48d6590f171a03" gracePeriod=30 Jul 08 19:43:21.980514 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:21.980480 2576 generic.go:356] "Generic (PLEG): container finished" podID="80c91fd0-b7e6-42b8-83a8-30118600715e" containerID="0b54eb70bcb0db8561bcf1250d2abfaf1dbe4e80b8c50b091b48d6590f171a03" exitCode=0 Jul 08 19:43:21.980979 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:21.980530 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" event={"ID":"80c91fd0-b7e6-42b8-83a8-30118600715e","Type":"ContainerDied","Data":"0b54eb70bcb0db8561bcf1250d2abfaf1dbe4e80b8c50b091b48d6590f171a03"} Jul 08 19:43:22.091777 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.091747 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:22.230846 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.230758 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-podinfo\") pod \"80c91fd0-b7e6-42b8-83a8-30118600715e\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " Jul 08 19:43:22.230846 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.230799 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-socket\") pod \"80c91fd0-b7e6-42b8-83a8-30118600715e\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " Jul 08 19:43:22.230846 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.230816 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-token\") pod \"80c91fd0-b7e6-42b8-83a8-30118600715e\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " Jul 08 19:43:22.231122 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.230848 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-credential-socket\") pod \"80c91fd0-b7e6-42b8-83a8-30118600715e\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " Jul 08 19:43:22.231122 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.230900 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-data\") pod \"80c91fd0-b7e6-42b8-83a8-30118600715e\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " Jul 08 19:43:22.231122 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.230930 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c91fd0-b7e6-42b8-83a8-30118600715e-istiod-ca-cert\") pod \"80c91fd0-b7e6-42b8-83a8-30118600715e\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " Jul 08 19:43:22.231122 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.230962 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-certs\") pod \"80c91fd0-b7e6-42b8-83a8-30118600715e\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " Jul 08 19:43:22.231122 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.231003 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47whp\" (UniqueName: \"kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-kube-api-access-47whp\") pod \"80c91fd0-b7e6-42b8-83a8-30118600715e\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " Jul 08 19:43:22.231122 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.231034 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-envoy\") pod \"80c91fd0-b7e6-42b8-83a8-30118600715e\" (UID: \"80c91fd0-b7e6-42b8-83a8-30118600715e\") " Jul 08 19:43:22.231465 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.231151 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "80c91fd0-b7e6-42b8-83a8-30118600715e" (UID: "80c91fd0-b7e6-42b8-83a8-30118600715e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:22.231465 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.231200 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "80c91fd0-b7e6-42b8-83a8-30118600715e" (UID: "80c91fd0-b7e6-42b8-83a8-30118600715e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:22.231465 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.231210 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-data" (OuterVolumeSpecName: "istio-data") pod "80c91fd0-b7e6-42b8-83a8-30118600715e" (UID: "80c91fd0-b7e6-42b8-83a8-30118600715e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:22.231465 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.231404 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "80c91fd0-b7e6-42b8-83a8-30118600715e" (UID: "80c91fd0-b7e6-42b8-83a8-30118600715e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:22.231465 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.231452 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-socket\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:22.231742 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.231471 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-credential-socket\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:22.231742 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.231486 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-data\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:22.231742 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.231700 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c91fd0-b7e6-42b8-83a8-30118600715e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "80c91fd0-b7e6-42b8-83a8-30118600715e" (UID: "80c91fd0-b7e6-42b8-83a8-30118600715e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:43:22.233021 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.232998 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-token" (OuterVolumeSpecName: "istio-token") pod "80c91fd0-b7e6-42b8-83a8-30118600715e" (UID: "80c91fd0-b7e6-42b8-83a8-30118600715e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:43:22.233179 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.233163 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "80c91fd0-b7e6-42b8-83a8-30118600715e" (UID: "80c91fd0-b7e6-42b8-83a8-30118600715e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:22.233337 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.233323 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "80c91fd0-b7e6-42b8-83a8-30118600715e" (UID: "80c91fd0-b7e6-42b8-83a8-30118600715e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:43:22.233473 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.233449 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-kube-api-access-47whp" (OuterVolumeSpecName: "kube-api-access-47whp") pod "80c91fd0-b7e6-42b8-83a8-30118600715e" (UID: "80c91fd0-b7e6-42b8-83a8-30118600715e"). InnerVolumeSpecName "kube-api-access-47whp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:43:22.332716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.332677 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-podinfo\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:22.332716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.332709 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-token\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:22.332716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.332718 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c91fd0-b7e6-42b8-83a8-30118600715e-istiod-ca-cert\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:22.332716 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.332727 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-workload-certs\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:22.332962 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.332735 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47whp\" (UniqueName: \"kubernetes.io/projected/80c91fd0-b7e6-42b8-83a8-30118600715e-kube-api-access-47whp\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:22.332962 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.332743 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c91fd0-b7e6-42b8-83a8-30118600715e-istio-envoy\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:22.986978 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.986942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" event={"ID":"80c91fd0-b7e6-42b8-83a8-30118600715e","Type":"ContainerDied","Data":"99257bd385166971511f870b43965fa735a5c3abb072a451b951c356856c7579"} Jul 08 19:43:22.986978 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.986974 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9" Jul 08 19:43:22.987487 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:22.987014 2576 scope.go:117] "RemoveContainer" containerID="0b54eb70bcb0db8561bcf1250d2abfaf1dbe4e80b8c50b091b48d6590f171a03" Jul 08 19:43:23.030295 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.030261 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9"] Jul 08 19:43:23.060653 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.060624 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-cf47b599-openshift-default-6676b7b669-zrvk9"] Jul 08 19:43:23.133155 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.133120 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884"] Jul 08 19:43:23.133549 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.133521 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c91fd0-b7e6-42b8-83a8-30118600715e" containerName="istio-proxy" Jul 08 19:43:23.133549 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.133550 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c91fd0-b7e6-42b8-83a8-30118600715e" containerName="istio-proxy" Jul 08 19:43:23.133704 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.133621 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c91fd0-b7e6-42b8-83a8-30118600715e" containerName="istio-proxy" Jul 08 19:43:23.138072 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.138047 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.141416 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.141396 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ec12e5f8-openshift-default-dockercfg-6ll9g\"" Jul 08 19:43:23.151936 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.151911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884"] Jul 08 19:43:23.239056 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.238957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-certs\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.239056 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.238993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-data\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.239056 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.239014 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b876f76-e909-409d-93fc-0c048b03a4a6-istiod-ca-cert\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.239361 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.239067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-token\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.239361 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.239114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-socket\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.239361 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.239177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-envoy\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.239361 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.239207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5wt7\" (UniqueName: \"kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-kube-api-access-v5wt7\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.239361 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.239257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-credential-socket\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.239361 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.239295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-podinfo\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.268454 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.268421 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80c91fd0-b7e6-42b8-83a8-30118600715e" path="/var/lib/kubelet/pods/80c91fd0-b7e6-42b8-83a8-30118600715e/volumes" Jul 08 19:43:23.340586 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-certs\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.340586 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-data\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.340815 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b876f76-e909-409d-93fc-0c048b03a4a6-istiod-ca-cert\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.340815 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-token\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.340815 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-socket\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.340815 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-envoy\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.340815 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5wt7\" (UniqueName: \"kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-kube-api-access-v5wt7\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.340815 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-credential-socket\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.341143 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-podinfo\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.341143 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340960 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-certs\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.341143 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.340970 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-data\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.341429 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.341401 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-credential-socket\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.341429 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.341429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b876f76-e909-409d-93fc-0c048b03a4a6-istiod-ca-cert\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.341635 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.341464 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-socket\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.343126 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.343106 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-envoy\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.343476 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.343455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-podinfo\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.350029 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.350003 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-token\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.350712 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.350684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5wt7\" (UniqueName: \"kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-kube-api-access-v5wt7\") pod \"e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.450118 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.450086 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:23.589483 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.589457 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884"] Jul 08 19:43:23.591074 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:43:23.591048 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b876f76_e909_409d_93fc_0c048b03a4a6.slice/crio-48caae131a8cf03269b6dc468eb1b101457c1232b07561dfdd3bc572fd7dfe31 WatchSource:0}: Error finding container 48caae131a8cf03269b6dc468eb1b101457c1232b07561dfdd3bc572fd7dfe31: Status 404 returned error can't find the container with id 48caae131a8cf03269b6dc468eb1b101457c1232b07561dfdd3bc572fd7dfe31 Jul 08 19:43:23.593278 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.593244 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:43:23.593415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.593307 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:43:23.593415 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.593334 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:43:23.992925 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.992892 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" event={"ID":"9b876f76-e909-409d-93fc-0c048b03a4a6","Type":"ContainerStarted","Data":"89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b"} Jul 08 19:43:23.993268 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:23.992953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" event={"ID":"9b876f76-e909-409d-93fc-0c048b03a4a6","Type":"ContainerStarted","Data":"48caae131a8cf03269b6dc468eb1b101457c1232b07561dfdd3bc572fd7dfe31"} Jul 08 19:43:24.017378 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:24.017313 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" podStartSLOduration=1.017294476 podStartE2EDuration="1.017294476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:43:24.016714751 +0000 UTC m=+1921.319018918" watchObservedRunningTime="2026-07-08 19:43:24.017294476 +0000 UTC m=+1921.319598645" Jul 08 19:43:24.451154 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:24.451118 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:24.456782 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:24.456758 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:24.998169 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:24.998142 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:24.999249 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:24.999233 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:30.414985 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:30.414952 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884"] Jul 08 19:43:30.415385 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:30.415137 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" podUID="9b876f76-e909-409d-93fc-0c048b03a4a6" containerName="istio-proxy" containerID="cri-o://89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b" gracePeriod=30 Jul 08 19:43:35.662094 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.662066 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:35.745251 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745167 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5wt7\" (UniqueName: \"kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-kube-api-access-v5wt7\") pod \"9b876f76-e909-409d-93fc-0c048b03a4a6\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " Jul 08 19:43:35.745251 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745205 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-token\") pod \"9b876f76-e909-409d-93fc-0c048b03a4a6\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " Jul 08 19:43:35.745251 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745230 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-certs\") pod \"9b876f76-e909-409d-93fc-0c048b03a4a6\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " Jul 08 19:43:35.745251 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745250 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b876f76-e909-409d-93fc-0c048b03a4a6-istiod-ca-cert\") pod \"9b876f76-e909-409d-93fc-0c048b03a4a6\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " Jul 08 19:43:35.745578 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745268 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-socket\") pod \"9b876f76-e909-409d-93fc-0c048b03a4a6\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " Jul 08 19:43:35.745578 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745284 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-credential-socket\") pod \"9b876f76-e909-409d-93fc-0c048b03a4a6\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " Jul 08 19:43:35.745578 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745327 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-data\") pod \"9b876f76-e909-409d-93fc-0c048b03a4a6\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " Jul 08 19:43:35.745578 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745368 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-podinfo\") pod \"9b876f76-e909-409d-93fc-0c048b03a4a6\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " Jul 08 19:43:35.745578 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745415 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-envoy\") pod \"9b876f76-e909-409d-93fc-0c048b03a4a6\" (UID: \"9b876f76-e909-409d-93fc-0c048b03a4a6\") " Jul 08 19:43:35.745793 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745608 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b876f76-e909-409d-93fc-0c048b03a4a6" (UID: "9b876f76-e909-409d-93fc-0c048b03a4a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:35.745793 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745656 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b876f76-e909-409d-93fc-0c048b03a4a6" (UID: "9b876f76-e909-409d-93fc-0c048b03a4a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:35.745793 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745679 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b876f76-e909-409d-93fc-0c048b03a4a6" (UID: "9b876f76-e909-409d-93fc-0c048b03a4a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:35.745793 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.745715 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b876f76-e909-409d-93fc-0c048b03a4a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b876f76-e909-409d-93fc-0c048b03a4a6" (UID: "9b876f76-e909-409d-93fc-0c048b03a4a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:43:35.746096 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.746070 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b876f76-e909-409d-93fc-0c048b03a4a6" (UID: "9b876f76-e909-409d-93fc-0c048b03a4a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:35.747502 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.747476 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-kube-api-access-v5wt7" (OuterVolumeSpecName: "kube-api-access-v5wt7") pod "9b876f76-e909-409d-93fc-0c048b03a4a6" (UID: "9b876f76-e909-409d-93fc-0c048b03a4a6"). InnerVolumeSpecName "kube-api-access-v5wt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:43:35.747615 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.747588 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b876f76-e909-409d-93fc-0c048b03a4a6" (UID: "9b876f76-e909-409d-93fc-0c048b03a4a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:43:35.747615 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.747607 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b876f76-e909-409d-93fc-0c048b03a4a6" (UID: "9b876f76-e909-409d-93fc-0c048b03a4a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:43:35.747705 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.747666 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b876f76-e909-409d-93fc-0c048b03a4a6" (UID: "9b876f76-e909-409d-93fc-0c048b03a4a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:43:35.846123 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.846088 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-data\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:35.846123 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.846118 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-podinfo\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:35.846123 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.846127 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-envoy\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:35.846370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.846136 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5wt7\" (UniqueName: \"kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-kube-api-access-v5wt7\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:35.846370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.846148 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b876f76-e909-409d-93fc-0c048b03a4a6-istio-token\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:35.846370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.846156 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-certs\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:35.846370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.846164 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b876f76-e909-409d-93fc-0c048b03a4a6-istiod-ca-cert\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:35.846370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.846171 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-workload-socket\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:35.846370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:35.846180 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b876f76-e909-409d-93fc-0c048b03a4a6-credential-socket\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:43:36.056941 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:36.056909 2576 generic.go:356] "Generic (PLEG): container finished" podID="9b876f76-e909-409d-93fc-0c048b03a4a6" containerID="89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b" exitCode=0 Jul 08 19:43:36.057107 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:36.056958 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" Jul 08 19:43:36.057107 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:36.056993 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" event={"ID":"9b876f76-e909-409d-93fc-0c048b03a4a6","Type":"ContainerDied","Data":"89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b"} Jul 08 19:43:36.057107 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:36.057048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884" event={"ID":"9b876f76-e909-409d-93fc-0c048b03a4a6","Type":"ContainerDied","Data":"48caae131a8cf03269b6dc468eb1b101457c1232b07561dfdd3bc572fd7dfe31"} Jul 08 19:43:36.057107 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:36.057080 2576 scope.go:117] "RemoveContainer" containerID="89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b" Jul 08 19:43:36.066526 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:36.066508 2576 scope.go:117] "RemoveContainer" containerID="89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b" Jul 08 19:43:36.066781 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:43:36.066763 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b\": container with ID starting with 89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b not found: ID does not exist" containerID="89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b" Jul 08 19:43:36.066827 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:36.066789 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b"} err="failed to get container status \"89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b\": rpc error: code = NotFound desc = could not find container \"89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b\": container with ID starting with 89be2670bf8a5906250a7dec1058919af85e2babb394014bf109755bfa392d0b not found: ID does not exist" Jul 08 19:43:36.079367 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:36.079332 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884"] Jul 08 19:43:36.083016 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:36.082995 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ec12e5f8-openshift-default-565f788ff7-rm884"] Jul 08 19:43:37.268549 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:43:37.268517 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b876f76-e909-409d-93fc-0c048b03a4a6" path="/var/lib/kubelet/pods/9b876f76-e909-409d-93fc-0c048b03a4a6/volumes" Jul 08 19:46:23.311331 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:46:23.311307 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:46:23.314780 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:46:23.314762 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:50:19.019853 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.019782 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr"] Jul 08 19:50:19.020321 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.020135 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b876f76-e909-409d-93fc-0c048b03a4a6" containerName="istio-proxy" Jul 08 19:50:19.020321 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.020146 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b876f76-e909-409d-93fc-0c048b03a4a6" containerName="istio-proxy" Jul 08 19:50:19.020321 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.020217 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b876f76-e909-409d-93fc-0c048b03a4a6" containerName="istio-proxy" Jul 08 19:50:19.023321 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.023297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.028036 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.028012 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0df657-openshift-default-dockercfg-757zz\"" Jul 08 19:50:19.039321 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.039298 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr"] Jul 08 19:50:19.118715 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.118685 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-istio-token\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.118715 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.118726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-data\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.118965 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.118745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63273387-93ce-4a35-8163-31151f643a75-istiod-ca-cert\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.118965 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.118843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-socket\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.118965 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.118871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63273387-93ce-4a35-8163-31151f643a75-istio-podinfo\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.118965 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.118894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skr6w\" (UniqueName: \"kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-kube-api-access-skr6w\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.119250 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.118968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-envoy\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.119250 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.119010 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-certs\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.119250 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.119043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-credential-socket\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220001 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.219959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-data\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220001 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.219998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63273387-93ce-4a35-8163-31151f643a75-istiod-ca-cert\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220261 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-socket\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220261 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63273387-93ce-4a35-8163-31151f643a75-istio-podinfo\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220261 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skr6w\" (UniqueName: \"kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-kube-api-access-skr6w\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220475 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-envoy\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220475 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-certs\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220475 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-credential-socket\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220475 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220339 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-data\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220475 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-istio-token\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220806 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220527 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-socket\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220806 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220736 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-certs\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220806 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220747 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63273387-93ce-4a35-8163-31151f643a75-istiod-ca-cert\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.220806 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.220784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-credential-socket\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.222587 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.222563 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-envoy\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.222919 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.222902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63273387-93ce-4a35-8163-31151f643a75-istio-podinfo\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.229113 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.229094 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skr6w\" (UniqueName: \"kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-kube-api-access-skr6w\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.229223 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.229179 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-istio-token\") pod \"e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.334687 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.334657 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:19.481396 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.481366 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr"] Jul 08 19:50:19.488014 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.487977 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:50:19.488131 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.488066 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:50:19.488131 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:19.488114 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:50:20.121191 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:20.121157 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" event={"ID":"63273387-93ce-4a35-8163-31151f643a75","Type":"ContainerStarted","Data":"88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786"} Jul 08 19:50:20.121677 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:20.121210 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" event={"ID":"63273387-93ce-4a35-8163-31151f643a75","Type":"ContainerStarted","Data":"539d1aede8f2719c7d2430e2fe06a53601cad0cafc61acf24a07cf517c362390"} Jul 08 19:50:20.143360 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:20.143303 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" podStartSLOduration=2.14328679 podStartE2EDuration="2.14328679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:50:20.142716952 +0000 UTC m=+2337.445021119" watchObservedRunningTime="2026-07-08 19:50:20.14328679 +0000 UTC m=+2337.445590958" Jul 08 19:50:20.334945 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:20.334906 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:20.339988 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:20.339963 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:21.126598 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:21.126559 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:21.127499 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:21.127474 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:37.168548 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:37.168514 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 19:50:37.168960 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:37.168566 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 19:50:38.214067 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:38.213990 2576 generic.go:356] "Generic (PLEG): container finished" podID="63273387-93ce-4a35-8163-31151f643a75" containerID="88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786" exitCode=0 Jul 08 19:50:38.214531 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:38.214064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" event={"ID":"63273387-93ce-4a35-8163-31151f643a75","Type":"ContainerDied","Data":"88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786"} Jul 08 19:50:38.214531 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:38.214466 2576 scope.go:117] "RemoveContainer" containerID="88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786" Jul 08 19:50:38.215462 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:38.215433 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:50:38.215579 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:38.215523 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:50:38.215655 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:38.215578 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:50:39.220860 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:39.220824 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" event={"ID":"63273387-93ce-4a35-8163-31151f643a75","Type":"ContainerStarted","Data":"00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f"} Jul 08 19:50:39.335821 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:39.335782 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:39.336018 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:39.335919 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 08 19:50:39.336018 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:39.335964 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 08 19:50:40.335980 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:40.335934 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 08 19:50:40.336476 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:40.336013 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 08 19:50:41.335818 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:41.335767 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 08 19:50:41.336031 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:41.335864 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 08 19:50:42.335384 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:42.335320 2576 patch_prober.go:28] interesting pod/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 08 19:50:42.335593 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:42.335408 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 08 19:50:43.338824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:43.338786 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:43.339229 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:43.339161 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:50:43.339874 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:50:43.339854 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:51:23.344847 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:51:23.344820 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:51:23.349357 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:51:23.349325 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hr5zg_36b08b02-b9a1-4d19-836a-5cc004fb0c0d/ovn-acl-logging/0.log" Jul 08 19:53:46.440566 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:46.440489 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr"] Jul 08 19:53:46.441172 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:46.440729 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" containerID="cri-o://00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f" gracePeriod=30 Jul 08 19:53:51.690853 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.690826 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:53:51.855544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855516 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-data\") pod \"63273387-93ce-4a35-8163-31151f643a75\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " Jul 08 19:53:51.855544 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855551 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skr6w\" (UniqueName: \"kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-kube-api-access-skr6w\") pod \"63273387-93ce-4a35-8163-31151f643a75\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " Jul 08 19:53:51.855824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855574 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-credential-socket\") pod \"63273387-93ce-4a35-8163-31151f643a75\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " Jul 08 19:53:51.855824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855595 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-certs\") pod \"63273387-93ce-4a35-8163-31151f643a75\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " Jul 08 19:53:51.855824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855614 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-envoy\") pod \"63273387-93ce-4a35-8163-31151f643a75\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " Jul 08 19:53:51.855824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855664 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-socket\") pod \"63273387-93ce-4a35-8163-31151f643a75\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " Jul 08 19:53:51.855824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855711 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63273387-93ce-4a35-8163-31151f643a75-istiod-ca-cert\") pod \"63273387-93ce-4a35-8163-31151f643a75\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " Jul 08 19:53:51.855824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855736 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63273387-93ce-4a35-8163-31151f643a75-istio-podinfo\") pod \"63273387-93ce-4a35-8163-31151f643a75\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " Jul 08 19:53:51.855824 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855773 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-istio-token\") pod \"63273387-93ce-4a35-8163-31151f643a75\" (UID: \"63273387-93ce-4a35-8163-31151f643a75\") " Jul 08 19:53:51.856174 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855888 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63273387-93ce-4a35-8163-31151f643a75" (UID: "63273387-93ce-4a35-8163-31151f643a75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:53:51.856174 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.855963 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63273387-93ce-4a35-8163-31151f643a75" (UID: "63273387-93ce-4a35-8163-31151f643a75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:53:51.856174 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.856047 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-credential-socket\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:53:51.856174 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.856067 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-certs\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:53:51.856174 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.856092 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63273387-93ce-4a35-8163-31151f643a75" (UID: "63273387-93ce-4a35-8163-31151f643a75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:53:51.856450 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.856415 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63273387-93ce-4a35-8163-31151f643a75-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63273387-93ce-4a35-8163-31151f643a75" (UID: "63273387-93ce-4a35-8163-31151f643a75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:53:51.856639 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.856596 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-data" (OuterVolumeSpecName: "istio-data") pod "63273387-93ce-4a35-8163-31151f643a75" (UID: "63273387-93ce-4a35-8163-31151f643a75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:53:51.857947 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.857919 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63273387-93ce-4a35-8163-31151f643a75-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63273387-93ce-4a35-8163-31151f643a75" (UID: "63273387-93ce-4a35-8163-31151f643a75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:53:51.858063 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.857943 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-kube-api-access-skr6w" (OuterVolumeSpecName: "kube-api-access-skr6w") pod "63273387-93ce-4a35-8163-31151f643a75" (UID: "63273387-93ce-4a35-8163-31151f643a75"). InnerVolumeSpecName "kube-api-access-skr6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:53:51.858238 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.858218 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63273387-93ce-4a35-8163-31151f643a75" (UID: "63273387-93ce-4a35-8163-31151f643a75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:53:51.858303 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.858236 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-istio-token" (OuterVolumeSpecName: "istio-token") pod "63273387-93ce-4a35-8163-31151f643a75" (UID: "63273387-93ce-4a35-8163-31151f643a75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:53:51.956836 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.956805 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-data\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:53:51.956836 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.956830 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skr6w\" (UniqueName: \"kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-kube-api-access-skr6w\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:53:51.956836 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.956840 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-istio-envoy\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:53:51.957061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.956851 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63273387-93ce-4a35-8163-31151f643a75-workload-socket\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:53:51.957061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.956858 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63273387-93ce-4a35-8163-31151f643a75-istiod-ca-cert\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:53:51.957061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.956868 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63273387-93ce-4a35-8163-31151f643a75-istio-podinfo\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:53:51.957061 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:51.956876 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63273387-93ce-4a35-8163-31151f643a75-istio-token\") on node \"ip-10-0-128-188.ec2.internal\" DevicePath \"\"" Jul 08 19:53:52.235576 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.235488 2576 generic.go:356] "Generic (PLEG): container finished" podID="63273387-93ce-4a35-8163-31151f643a75" containerID="00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f" exitCode=0 Jul 08 19:53:52.235576 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.235554 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" Jul 08 19:53:52.235822 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.235575 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" event={"ID":"63273387-93ce-4a35-8163-31151f643a75","Type":"ContainerDied","Data":"00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f"} Jul 08 19:53:52.235822 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.235672 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr" event={"ID":"63273387-93ce-4a35-8163-31151f643a75","Type":"ContainerDied","Data":"539d1aede8f2719c7d2430e2fe06a53601cad0cafc61acf24a07cf517c362390"} Jul 08 19:53:52.235822 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.235713 2576 scope.go:117] "RemoveContainer" containerID="00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f" Jul 08 19:53:52.245793 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.245779 2576 scope.go:117] "RemoveContainer" containerID="88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786" Jul 08 19:53:52.254006 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.253989 2576 scope.go:117] "RemoveContainer" containerID="00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f" Jul 08 19:53:52.254266 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:53:52.254249 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f\": container with ID starting with 00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f not found: ID does not exist" containerID="00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f" Jul 08 19:53:52.254330 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.254273 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f"} err="failed to get container status \"00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f\": rpc error: code = NotFound desc = could not find container \"00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f\": container with ID starting with 00db2f89e5f19a1799faa6ac3064fa4aa2a7c98d9d47e6cff657c372b885d15f not found: ID does not exist" Jul 08 19:53:52.254330 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.254291 2576 scope.go:117] "RemoveContainer" containerID="88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786" Jul 08 19:53:52.254532 ip-10-0-128-188 kubenswrapper[2576]: E0708 19:53:52.254516 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786\": container with ID starting with 88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786 not found: ID does not exist" containerID="88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786" Jul 08 19:53:52.254582 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.254536 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786"} err="failed to get container status \"88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786\": rpc error: code = NotFound desc = could not find container \"88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786\": container with ID starting with 88079fbbfd7930da200deacfe372a7ae3f30c5a91d23a6e0c421b3ab78281786 not found: ID does not exist" Jul 08 19:53:52.268084 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.268063 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr"] Jul 08 19:53:52.273772 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:52.273749 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0df657-openshift-default-7798798b5c-8hwpr"] Jul 08 19:53:53.267899 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:53.267867 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63273387-93ce-4a35-8163-31151f643a75" path="/var/lib/kubelet/pods/63273387-93ce-4a35-8163-31151f643a75/volumes" Jul 08 19:53:55.395172 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:53:55.395140 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65dc567574-m7dhm_d0ff258c-cca7-4501-8432-e7c7318607b2/authorino/0.log" Jul 08 19:54:00.118696 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:00.118666 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6zsf5_fca884ca-d632-498d-849d-b76609c5f1b8/manager/0.log" Jul 08 19:54:00.236933 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:00.236905 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2d2pq_2340417d-8d50-4bc8-b70a-5aa2f246432c/manager/0.log" Jul 08 19:54:00.484319 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:00.484241 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xtlqw_b863103a-45b1-46b7-8ad3-1ae10ebd8033/server/0.log" Jul 08 19:54:00.603188 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:00.603156 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v7f6g_5f0982f4-2a41-43fd-aa92-659927d24ad9/manager/0.log" Jul 08 19:54:00.877861 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:00.877836 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dc4d9889-9ccgz_a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9/copy-manifests/0.log" Jul 08 19:54:00.884427 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:00.884407 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dc4d9889-9ccgz_a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9/manager/0.log" Jul 08 19:54:01.792431 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:01.792399 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd_e8c49737-2a60-45a5-a337-9cd86ae76e58/pull/0.log" Jul 08 19:54:01.800321 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:01.800293 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd_e8c49737-2a60-45a5-a337-9cd86ae76e58/extract/0.log" Jul 08 19:54:01.810835 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:01.810817 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd_e8c49737-2a60-45a5-a337-9cd86ae76e58/util/0.log" Jul 08 19:54:01.941313 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:01.941291 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds_3317a351-ff9f-4255-a3c4-282552c4c483/util/0.log" Jul 08 19:54:01.950715 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:01.950697 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds_3317a351-ff9f-4255-a3c4-282552c4c483/pull/0.log" Jul 08 19:54:01.963608 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:01.963587 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds_3317a351-ff9f-4255-a3c4-282552c4c483/extract/0.log" Jul 08 19:54:02.080506 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:02.080438 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8_5538bfe3-3461-414b-bc7e-dfef1e582edb/util/0.log" Jul 08 19:54:02.094135 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:02.094114 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8_5538bfe3-3461-414b-bc7e-dfef1e582edb/pull/0.log" Jul 08 19:54:02.102999 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:02.102979 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8_5538bfe3-3461-414b-bc7e-dfef1e582edb/extract/0.log" Jul 08 19:54:02.222973 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:02.222941 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd_40918d12-8097-4ba3-b1eb-d13ec143f851/util/0.log" Jul 08 19:54:02.233106 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:02.233085 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd_40918d12-8097-4ba3-b1eb-d13ec143f851/pull/0.log" Jul 08 19:54:02.240736 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:02.240717 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd_40918d12-8097-4ba3-b1eb-d13ec143f851/extract/0.log" Jul 08 19:54:02.359374 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:02.359258 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65dc567574-m7dhm_d0ff258c-cca7-4501-8432-e7c7318607b2/authorino/0.log" Jul 08 19:54:02.495370 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:02.495319 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tk7cv_8941f64f-6a00-43c4-8f5d-9315478512bd/manager/0.log" Jul 08 19:54:02.855311 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:02.855265 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-v8b52_5cfd158d-de8b-4588-a011-a67ce68db14e/registry-server/0.log" Jul 08 19:54:02.999232 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:02.999205 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-47rb2_cde2715b-b16b-4df2-87bc-c6d94b83736f/manager/0.log" Jul 08 19:54:03.278236 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:03.278150 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gc8cz_79c731d1-ded9-44d4-9b6f-035c1d0c55bc/manager/0.log" Jul 08 19:54:03.629264 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:03.629228 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cgrkv_422aebe4-bd88-43f9-8193-e210f4c4b0b3/istio-proxy/0.log" Jul 08 19:54:03.871189 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:03.871155 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d9fdb5547-lvgpw_e475a700-abbe-41f4-984b-5fcfbbb0ee27/kube-auth-proxy/0.log" Jul 08 19:54:04.499858 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:04.499829 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54f56f5b6f-8l27s_94079af4-8a24-474d-871d-8bf792b247c4/router/0.log" Jul 08 19:54:04.851664 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:04.851635 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-f5pfn_24bbd019-1168-4202-9c54-e441d958c545/main/0.log" Jul 08 19:54:05.203208 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:05.203109 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbr65xm_5d7eb4f8-b60b-42ab-a1c6-d5a1bc4c4dbf/main/0.log" Jul 08 19:54:09.924541 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.924503 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4ch4n/must-gather-jzmst"] Jul 08 19:54:09.924931 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.924920 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" Jul 08 19:54:09.924974 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.924936 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" Jul 08 19:54:09.924974 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.924950 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" Jul 08 19:54:09.924974 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.924955 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" Jul 08 19:54:09.925069 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.925017 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" Jul 08 19:54:09.925127 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.925116 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63273387-93ce-4a35-8163-31151f643a75" containerName="istio-proxy" Jul 08 19:54:09.927161 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.927146 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ch4n/must-gather-jzmst" Jul 08 19:54:09.935086 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.935060 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ch4n\"/\"openshift-service-ca.crt\"" Jul 08 19:54:09.935086 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.935070 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4ch4n\"/\"kube-root-ca.crt\"" Jul 08 19:54:09.943810 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.943783 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ch4n/must-gather-jzmst"] Jul 08 19:54:09.996426 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.996403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d747a7e3-5fb9-4a93-956f-7874ae9eb738-must-gather-output\") pod \"must-gather-jzmst\" (UID: \"d747a7e3-5fb9-4a93-956f-7874ae9eb738\") " pod="openshift-must-gather-4ch4n/must-gather-jzmst" Jul 08 19:54:09.996621 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:09.996472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6h7l\" (UniqueName: \"kubernetes.io/projected/d747a7e3-5fb9-4a93-956f-7874ae9eb738-kube-api-access-n6h7l\") pod \"must-gather-jzmst\" (UID: \"d747a7e3-5fb9-4a93-956f-7874ae9eb738\") " pod="openshift-must-gather-4ch4n/must-gather-jzmst" Jul 08 19:54:10.097884 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:10.097840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6h7l\" (UniqueName: \"kubernetes.io/projected/d747a7e3-5fb9-4a93-956f-7874ae9eb738-kube-api-access-n6h7l\") pod \"must-gather-jzmst\" (UID: \"d747a7e3-5fb9-4a93-956f-7874ae9eb738\") " pod="openshift-must-gather-4ch4n/must-gather-jzmst" Jul 08 19:54:10.098053 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:10.097933 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d747a7e3-5fb9-4a93-956f-7874ae9eb738-must-gather-output\") pod \"must-gather-jzmst\" (UID: \"d747a7e3-5fb9-4a93-956f-7874ae9eb738\") " pod="openshift-must-gather-4ch4n/must-gather-jzmst" Jul 08 19:54:10.098262 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:10.098241 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d747a7e3-5fb9-4a93-956f-7874ae9eb738-must-gather-output\") pod \"must-gather-jzmst\" (UID: \"d747a7e3-5fb9-4a93-956f-7874ae9eb738\") " pod="openshift-must-gather-4ch4n/must-gather-jzmst" Jul 08 19:54:10.112378 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:10.112340 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6h7l\" (UniqueName: \"kubernetes.io/projected/d747a7e3-5fb9-4a93-956f-7874ae9eb738-kube-api-access-n6h7l\") pod \"must-gather-jzmst\" (UID: \"d747a7e3-5fb9-4a93-956f-7874ae9eb738\") " pod="openshift-must-gather-4ch4n/must-gather-jzmst" Jul 08 19:54:10.236308 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:10.236209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ch4n/must-gather-jzmst" Jul 08 19:54:10.380617 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:10.380587 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ch4n/must-gather-jzmst"] Jul 08 19:54:10.381489 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:54:10.381463 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd747a7e3_5fb9_4a93_956f_7874ae9eb738.slice/crio-e79609844bfad6b06b4d2b4d97d2e39b1375294e43566a1cc62837c13c8ea505 WatchSource:0}: Error finding container e79609844bfad6b06b4d2b4d97d2e39b1375294e43566a1cc62837c13c8ea505: Status 404 returned error can't find the container with id e79609844bfad6b06b4d2b4d97d2e39b1375294e43566a1cc62837c13c8ea505 Jul 08 19:54:10.383131 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:10.383114 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:54:11.349166 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:11.349070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ch4n/must-gather-jzmst" event={"ID":"d747a7e3-5fb9-4a93-956f-7874ae9eb738","Type":"ContainerStarted","Data":"6846e09b071fecdf7c3ca954fd8404774d9b259e4e292bab0a83c43d41a8e901"} Jul 08 19:54:11.349166 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:11.349111 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ch4n/must-gather-jzmst" event={"ID":"d747a7e3-5fb9-4a93-956f-7874ae9eb738","Type":"ContainerStarted","Data":"580a2adc0cca330fb14899c213c44776dedb8aae0e27a09ee42a468d9615774c"} Jul 08 19:54:11.349166 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:11.349125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ch4n/must-gather-jzmst" event={"ID":"d747a7e3-5fb9-4a93-956f-7874ae9eb738","Type":"ContainerStarted","Data":"e79609844bfad6b06b4d2b4d97d2e39b1375294e43566a1cc62837c13c8ea505"} Jul 08 19:54:11.370195 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:11.370127 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4ch4n/must-gather-jzmst" podStartSLOduration=1.720675697 podStartE2EDuration="2.370103909s" totalImagesPullingTime="649.428212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:54:11.36716927 +0000 UTC m=+2568.669473439" watchObservedRunningTime="2026-07-08 19:54:11.370103909 +0000 UTC m=+2568.672408078" Jul 08 19:54:13.230743 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:13.230688 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6qs6k_5b982417-f3a9-43ff-9a76-920bd553524e/global-pull-secret-syncer/0.log" Jul 08 19:54:13.974378 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:13.974317 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zqct5_59a6b211-27a3-4086-914f-cfb2a1418f9b/konnectivity-agent/0.log" Jul 08 19:54:14.061742 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:14.061706 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-188.ec2.internal_5117d2d045a1c775b90c668c570fae47/haproxy/0.log" Jul 08 19:54:17.850072 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:17.850044 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd_e8c49737-2a60-45a5-a337-9cd86ae76e58/extract/0.log" Jul 08 19:54:17.921617 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:17.921584 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd_e8c49737-2a60-45a5-a337-9cd86ae76e58/util/0.log" Jul 08 19:54:17.979849 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:17.979823 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7599d4dd_e8c49737-2a60-45a5-a337-9cd86ae76e58/pull/0.log" Jul 08 19:54:18.045767 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:18.045735 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds_3317a351-ff9f-4255-a3c4-282552c4c483/extract/0.log" Jul 08 19:54:18.115631 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:18.115543 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds_3317a351-ff9f-4255-a3c4-282552c4c483/util/0.log" Jul 08 19:54:18.181655 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:18.181616 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z67ds_3317a351-ff9f-4255-a3c4-282552c4c483/pull/0.log" Jul 08 19:54:18.291838 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:18.291805 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8_5538bfe3-3461-414b-bc7e-dfef1e582edb/extract/0.log" Jul 08 19:54:18.378722 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:18.378639 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8_5538bfe3-3461-414b-bc7e-dfef1e582edb/util/0.log" Jul 08 19:54:18.468636 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:18.468611 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73jkzs8_5538bfe3-3461-414b-bc7e-dfef1e582edb/pull/0.log" Jul 08 19:54:18.563707 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:18.563665 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd_40918d12-8097-4ba3-b1eb-d13ec143f851/extract/0.log" Jul 08 19:54:18.642171 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:18.642098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd_40918d12-8097-4ba3-b1eb-d13ec143f851/util/0.log" Jul 08 19:54:18.727759 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:18.727725 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1qbfdd_40918d12-8097-4ba3-b1eb-d13ec143f851/pull/0.log" Jul 08 19:54:19.094366 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:19.094120 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65dc567574-m7dhm_d0ff258c-cca7-4501-8432-e7c7318607b2/authorino/0.log" Jul 08 19:54:19.159545 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:19.159510 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tk7cv_8941f64f-6a00-43c4-8f5d-9315478512bd/manager/0.log" Jul 08 19:54:19.395852 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:19.395743 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-v8b52_5cfd158d-de8b-4588-a011-a67ce68db14e/registry-server/0.log" Jul 08 19:54:19.538910 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:19.538878 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-47rb2_cde2715b-b16b-4df2-87bc-c6d94b83736f/manager/0.log" Jul 08 19:54:19.731301 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:19.731210 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gc8cz_79c731d1-ded9-44d4-9b6f-035c1d0c55bc/manager/0.log" Jul 08 19:54:22.440299 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:22.440267 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29ws8_a1eb4547-ca65-447a-a218-3fc40397407c/node-exporter/0.log" Jul 08 19:54:22.551301 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:22.551267 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29ws8_a1eb4547-ca65-447a-a218-3fc40397407c/kube-rbac-proxy/0.log" Jul 08 19:54:22.636478 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:22.636446 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-29ws8_a1eb4547-ca65-447a-a218-3fc40397407c/init-textfile/0.log" Jul 08 19:54:22.902157 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:22.902124 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq"] Jul 08 19:54:22.908211 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:22.908181 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:22.911482 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:22.911457 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4ch4n\"/\"default-dockercfg-ml2jp\"" Jul 08 19:54:22.928837 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:22.928807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq"] Jul 08 19:54:23.022736 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.022704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-lib-modules\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.022922 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.022750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-podres\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.022922 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.022820 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-proc\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.022922 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.022860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9j7r\" (UniqueName: \"kubernetes.io/projected/8333b846-e06f-4a0b-a8d7-ad09c81179e1-kube-api-access-g9j7r\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.023034 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.022944 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-sys\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.124315 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.124278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-sys\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.124486 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.124375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-lib-modules\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.124486 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.124424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-podres\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.124486 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.124393 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-sys\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.124486 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.124462 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-proc\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.124640 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.124513 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-proc\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.124640 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.124532 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-lib-modules\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.124640 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.124545 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8333b846-e06f-4a0b-a8d7-ad09c81179e1-podres\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.124640 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.124597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9j7r\" (UniqueName: \"kubernetes.io/projected/8333b846-e06f-4a0b-a8d7-ad09c81179e1-kube-api-access-g9j7r\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.144630 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.144600 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9j7r\" (UniqueName: \"kubernetes.io/projected/8333b846-e06f-4a0b-a8d7-ad09c81179e1-kube-api-access-g9j7r\") pod \"perf-node-gather-daemonset-hlkjq\" (UID: \"8333b846-e06f-4a0b-a8d7-ad09c81179e1\") " pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.226373 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.226273 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4ch4n\"/\"default-dockercfg-ml2jp\"" Jul 08 19:54:23.232353 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.232316 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:23.405922 ip-10-0-128-188 kubenswrapper[2576]: W0708 19:54:23.405888 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8333b846_e06f_4a0b_a8d7_ad09c81179e1.slice/crio-5307e9775ecab35efeae32074c83da9ef4827a982846f7278290ae99a6e6b186 WatchSource:0}: Error finding container 5307e9775ecab35efeae32074c83da9ef4827a982846f7278290ae99a6e6b186: Status 404 returned error can't find the container with id 5307e9775ecab35efeae32074c83da9ef4827a982846f7278290ae99a6e6b186 Jul 08 19:54:23.408134 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.408110 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq"] Jul 08 19:54:23.467367 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:23.467318 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" event={"ID":"8333b846-e06f-4a0b-a8d7-ad09c81179e1","Type":"ContainerStarted","Data":"5307e9775ecab35efeae32074c83da9ef4827a982846f7278290ae99a6e6b186"} Jul 08 19:54:24.475867 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:24.475788 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" event={"ID":"8333b846-e06f-4a0b-a8d7-ad09c81179e1","Type":"ContainerStarted","Data":"de1457704017c06b517089e75571ef62fe858506d35b3462bfdcbf991dbc30f3"} Jul 08 19:54:24.476238 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:24.475897 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:24.505831 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:24.505780 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" podStartSLOduration=2.505763791 podStartE2EDuration="2.505763791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:54:24.503082154 +0000 UTC m=+2581.805386334" watchObservedRunningTime="2026-07-08 19:54:24.505763791 +0000 UTC m=+2581.808067958" Jul 08 19:54:27.041713 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:27.041683 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-lljqn_27d50fcc-7031-4354-8db6-023f93cad601/download-server/0.log" Jul 08 19:54:27.799862 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:27.799836 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-gd8tc_a82840dc-4271-44e5-bd09-1092c7de4858/volume-data-source-validator/0.log" Jul 08 19:54:29.168524 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:29.168492 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8njxt_53d10eff-78be-422c-b7ce-13d970eca76d/dns/0.log" Jul 08 19:54:29.213632 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:29.213606 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8njxt_53d10eff-78be-422c-b7ce-13d970eca76d/kube-rbac-proxy/0.log" Jul 08 19:54:29.582789 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:29.582757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zzbhq_1d27b693-5d3b-4cd2-b7e4-9ab67506e2ca/dns-node-resolver/0.log" Jul 08 19:54:30.491197 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:30.491170 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4ch4n/perf-node-gather-daemonset-hlkjq" Jul 08 19:54:30.521134 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:30.521101 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lldql_d385dcd3-5579-42a7-af3d-c9f272c9a568/node-ca/0.log" Jul 08 19:54:32.034905 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:32.034871 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cgrkv_422aebe4-bd88-43f9-8193-e210f4c4b0b3/istio-proxy/0.log" Jul 08 19:54:32.406363 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:32.406321 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d9fdb5547-lvgpw_e475a700-abbe-41f4-984b-5fcfbbb0ee27/kube-auth-proxy/0.log" Jul 08 19:54:33.134845 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:33.134778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54f56f5b6f-8l27s_94079af4-8a24-474d-871d-8bf792b247c4/router/0.log" Jul 08 19:54:33.267482 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:33.267453 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cgrkv_422aebe4-bd88-43f9-8193-e210f4c4b0b3/istio-proxy/0.log" Jul 08 19:54:33.512436 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:33.512361 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d9fdb5547-lvgpw_e475a700-abbe-41f4-984b-5fcfbbb0ee27/kube-auth-proxy/0.log" Jul 08 19:54:34.009115 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:34.009088 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54f56f5b6f-8l27s_94079af4-8a24-474d-871d-8bf792b247c4/router/0.log" Jul 08 19:54:34.294590 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:34.294558 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pbwkx_3db481d0-9cdf-46be-a738-505e054ba95b/serve-healthcheck-canary/0.log" Jul 08 19:54:35.311209 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:35.311185 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4cgrkv_422aebe4-bd88-43f9-8193-e210f4c4b0b3/istio-proxy/0.log" Jul 08 19:54:35.607233 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:35.607205 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d9fdb5547-lvgpw_e475a700-abbe-41f4-984b-5fcfbbb0ee27/kube-auth-proxy/0.log" Jul 08 19:54:35.816261 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:35.816235 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4752_17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57/kube-rbac-proxy/0.log" Jul 08 19:54:35.941084 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:35.941008 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4752_17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57/exporter/0.log" Jul 08 19:54:36.109559 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:36.109533 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4752_17e39ded-a2a4-4c7e-a350-6ca7fe8b3d57/extractor/0.log" Jul 08 19:54:36.111894 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:36.111868 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54f56f5b6f-8l27s_94079af4-8a24-474d-871d-8bf792b247c4/router/0.log" Jul 08 19:54:38.730697 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:38.730671 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6zsf5_fca884ca-d632-498d-849d-b76609c5f1b8/manager/0.log" Jul 08 19:54:38.844750 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:38.844717 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-2d2pq_2340417d-8d50-4bc8-b70a-5aa2f246432c/manager/0.log" Jul 08 19:54:39.088992 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:39.088963 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-xtlqw_b863103a-45b1-46b7-8ad3-1ae10ebd8033/server/0.log" Jul 08 19:54:39.195959 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:39.195928 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v7f6g_5f0982f4-2a41-43fd-aa92-659927d24ad9/manager/0.log" Jul 08 19:54:39.404479 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:39.404401 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dc4d9889-9ccgz_a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9/manager/0.log" Jul 08 19:54:39.445090 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:39.445059 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dc4d9889-9ccgz_a1fe1b1d-1c1a-43f0-832b-8bc42e55d9d9/copy-manifests/0.log" Jul 08 19:54:40.899250 ip-10-0-128-188 kubenswrapper[2576]: I0708 19:54:40.899171 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pmw9s_c6cd16c0-9ea7-4836-a978-3629e2a12005/openshift-lws-operator/0.log"