Jul 08 19:42:07.670450 ip-10-0-131-22 systemd[1]: Starting Kubernetes Kubelet... Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.195115 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198931 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198947 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198953 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198957 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198961 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198964 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198972 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:42:09.252760 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198975 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198978 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198981 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198984 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198987 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198990 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198992 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198995 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.198998 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199000 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199003 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199006 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199009 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199011 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199014 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199016 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199019 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199021 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199025 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199027 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:42:09.253864 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199030 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199032 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199035 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199037 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199040 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199042 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199045 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199047 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199050 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199052 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199055 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199057 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199062 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199065 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199068 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199071 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199074 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199078 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199082 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:42:09.258187 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199084 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199087 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199089 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199092 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199094 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199096 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199099 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199102 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199104 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199107 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199109 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199112 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199114 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199117 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199119 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199122 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199124 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199127 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199130 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199133 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:42:09.258823 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199136 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199139 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199142 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199144 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199147 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199150 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199179 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199183 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199185 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199188 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199191 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199193 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199196 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199199 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199201 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199204 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199206 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199210 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199213 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199215 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:42:09.259880 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199218 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199221 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199223 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199226 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199229 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199231 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199234 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199238 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199241 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199244 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199247 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199250 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.199252 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200085 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200091 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200094 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200097 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200101 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200103 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:42:09.260531 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200113 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200118 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200122 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200125 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200128 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200131 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200134 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200137 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200139 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200142 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200145 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200148 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200150 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200153 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200176 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200179 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200182 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200184 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200187 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:42:09.261603 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200190 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200192 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200195 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200197 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200202 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200206 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200209 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200211 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200214 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200217 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200221 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200225 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200228 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200230 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200239 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200241 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200244 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200246 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200249 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:42:09.266150 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200252 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200255 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200257 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200259 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200262 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200265 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200267 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200270 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200273 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200275 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200278 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200280 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200283 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200287 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200289 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200292 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200294 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200297 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200300 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200302 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:42:09.266821 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200304 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200307 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200309 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200313 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200315 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200323 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200326 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200328 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200336 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200339 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200342 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200344 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200347 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200349 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200351 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200354 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200357 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200359 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200362 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200365 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:42:09.267469 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200367 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200370 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200372 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200375 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200377 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200379 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200382 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200385 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200387 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200389 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200392 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200394 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200397 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200400 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200402 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200886 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200893 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200896 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200899 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200903 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:42:09.268977 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200906 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200909 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200912 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200915 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200918 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200921 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200923 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200927 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200930 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200933 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200936 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200939 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200941 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200944 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200948 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200952 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200954 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200957 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200960 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200963 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:42:09.269624 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200966 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200969 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200971 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200974 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200976 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200980 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200984 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200986 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200989 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200992 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200994 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200997 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.200999 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201002 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201005 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201007 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201010 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201012 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201014 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:42:09.270472 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201020 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201023 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201025 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201028 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201031 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201033 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201036 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201038 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201041 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201044 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201046 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201049 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201051 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201054 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201056 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201059 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201061 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201064 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201066 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:42:09.324687 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201069 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201071 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201074 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201076 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201078 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201082 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201084 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201087 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201092 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201094 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201097 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201099 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201102 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201107 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201110 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201112 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201115 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201118 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201120 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201123 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:42:09.325326 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201125 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201128 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201130 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201133 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201135 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201138 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201140 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201143 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201145 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201148 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201152 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201168 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201171 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201175 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201177 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201181 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201261 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201272 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201283 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201290 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201302 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 19:42:09.326856 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201306 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201310 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201315 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201318 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201323 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201327 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201330 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201333 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201337 2572 flags.go:64] FLAG: --cgroup-root="" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201340 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201343 2572 flags.go:64] FLAG: --client-ca-file="" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201345 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201348 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201353 2572 flags.go:64] FLAG: --cluster-domain="" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201356 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201359 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201362 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201365 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201369 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201372 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201376 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201379 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201382 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201385 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 19:42:09.327494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201388 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201391 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201394 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201398 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201402 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201404 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201407 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201412 2572 flags.go:64] FLAG: --enable-server="true" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201415 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201422 2572 flags.go:64] FLAG: --event-burst="100" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201426 2572 flags.go:64] FLAG: --event-qps="50" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201429 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201433 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201436 2572 flags.go:64] FLAG: --eviction-hard="" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201440 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201443 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201446 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201449 2572 flags.go:64] FLAG: --eviction-soft="" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201452 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201456 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201459 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201462 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201465 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201468 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201471 2572 flags.go:64] FLAG: --feature-gates="" Jul 08 19:42:09.329017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201475 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201478 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201481 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201484 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201488 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201491 2572 flags.go:64] FLAG: --help="false" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201493 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-131-22.ec2.internal" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201496 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201499 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201502 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201506 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201509 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201512 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201515 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201519 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201522 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201525 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201529 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201532 2572 flags.go:64] FLAG: --kube-reserved="" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201536 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201539 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201542 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201545 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201548 2572 flags.go:64] FLAG: --lock-file="" Jul 08 19:42:09.329905 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201551 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201554 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201557 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201562 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201566 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201569 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201571 2572 flags.go:64] FLAG: --logging-format="text" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201574 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201577 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201581 2572 flags.go:64] FLAG: --manifest-url="" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201583 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201588 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201591 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201595 2572 flags.go:64] FLAG: --max-pods="110" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201598 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201601 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201604 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201607 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201610 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201613 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201615 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201623 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201628 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201631 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 19:42:09.330574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201634 2572 flags.go:64] FLAG: --pod-cidr="" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201637 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201644 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201647 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201652 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201655 2572 flags.go:64] FLAG: --port="10250" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201659 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201662 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04bf09d70d8d9883f" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201665 2572 flags.go:64] FLAG: --qos-reserved="" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201668 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201671 2572 flags.go:64] FLAG: --register-node="true" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201674 2572 flags.go:64] FLAG: --register-with-taints="" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201678 2572 flags.go:64] FLAG: --registry-burst="10" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201681 2572 flags.go:64] FLAG: --registry-qps="5" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201683 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201686 2572 flags.go:64] FLAG: --reserved-memory="" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201690 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201693 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201696 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201699 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201702 2572 flags.go:64] FLAG: --runonce="false" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201705 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201708 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201711 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201713 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 19:42:09.331468 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201716 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201719 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201722 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201725 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201728 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201732 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201735 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201738 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201741 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201744 2572 flags.go:64] FLAG: --system-cgroups="" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201747 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201754 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201756 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201759 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201763 2572 flags.go:64] FLAG: --tls-min-version="" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201766 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201769 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201772 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201775 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201778 2572 flags.go:64] FLAG: --v="2" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201782 2572 flags.go:64] FLAG: --version="false" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201787 2572 flags.go:64] FLAG: --vmodule="" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201791 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.201794 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201915 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:42:09.332343 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201919 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201922 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201925 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201928 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201931 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201933 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201936 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201939 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201941 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201944 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201947 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201949 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201954 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201956 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201959 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201961 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201964 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201966 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201971 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201975 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:42:09.333425 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201979 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201982 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201985 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201987 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201990 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201992 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201995 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.201998 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202000 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202003 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202005 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202008 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202011 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202013 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202016 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202018 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202021 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202025 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202028 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:42:09.334058 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202031 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202034 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202037 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202040 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202042 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202047 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202049 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202052 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202055 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202057 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202060 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202065 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202068 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202070 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202073 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202076 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202078 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202081 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202084 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202087 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:42:09.334694 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202090 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202092 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202095 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202097 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202100 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202102 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202105 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202108 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202110 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202112 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202115 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202117 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202120 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202122 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202125 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202132 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202135 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202139 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202141 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202144 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:42:09.335346 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202147 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202149 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202152 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202170 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202172 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202175 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202178 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202181 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202183 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202186 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202188 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202191 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202194 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202196 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202199 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202201 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202204 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202207 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:42:09.335984 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.202211 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.202958 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.213193 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.213214 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.213236 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.213242 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213310 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213315 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213318 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213321 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213324 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213330 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213333 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213336 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:42:09.336639 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213339 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213344 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213348 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213351 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213355 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213358 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213361 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213364 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213366 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213369 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213371 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213374 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213377 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213379 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213382 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213385 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213387 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213390 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213392 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213395 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:42:09.337073 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213398 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213401 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213404 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213406 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213409 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213411 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213414 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213416 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213419 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213424 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213426 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213431 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213434 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213437 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213440 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213442 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213447 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213450 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213453 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:42:09.338380 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213455 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213458 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213460 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213463 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213466 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213469 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213471 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213474 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213476 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213479 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213481 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213484 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213487 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213489 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213492 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213494 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213497 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213499 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213502 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213505 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:42:09.339018 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213507 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213510 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213514 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213517 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213520 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213523 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213526 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213528 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213531 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213536 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213538 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213541 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213544 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213547 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213549 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213552 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213555 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213557 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213560 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213562 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:42:09.340755 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213565 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213567 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213570 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213572 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213575 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213579 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213583 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213585 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213588 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213591 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213593 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213596 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.213602 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213727 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213734 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 19:42:09.341394 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213737 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213740 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213743 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213746 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213748 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213751 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213754 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213758 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213761 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213764 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213767 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213770 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213772 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213775 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213777 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213780 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213782 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213785 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213788 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213791 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 19:42:09.342488 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213793 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213803 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213805 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213808 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213810 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213813 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213816 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213818 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213821 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213823 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213826 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213830 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213833 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213835 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213838 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213841 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213844 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213846 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213848 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213852 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 19:42:09.343117 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213855 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213858 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213861 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213865 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213868 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213870 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213873 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213876 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213878 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213881 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213883 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213886 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213888 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213891 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213894 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213896 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213899 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213901 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213904 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 19:42:09.343775 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213907 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213912 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213915 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213918 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213921 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213924 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213927 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213930 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213932 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213935 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213938 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213942 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213946 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213949 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213952 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213955 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213959 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213962 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213965 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213968 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 19:42:09.344435 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213971 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213974 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213976 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213979 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213981 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213985 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213988 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213991 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213994 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.213997 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.214000 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.214002 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.214005 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.214007 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.214010 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.214013 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.214017 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 19:42:09.345108 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:09.214020 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.214025 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.214179 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.216467 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.218111 2572 server.go:1019] "Starting client certificate rotation" Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.218229 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.218282 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.221855 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.223134 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.233527 2572 log.go:25] "Validated CRI v1 runtime API" Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.236069 2572 log.go:25] "Validated CRI v1 image API" Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.237045 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 19:42:09.346650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.256125 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 19:42:09.347091 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.276596 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5twz4" Jul 08 19:42:09.347091 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.284357 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5twz4" Jul 08 19:42:09.410415 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.390209 2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c1e2f0d0-f907-4868-b4d2-d810c94046cc:/dev/nvme0n1p4 d111f11b-080f-48e0-8c1c-e16a8cce10e4:/dev/nvme0n1p3] Jul 08 19:42:09.410415 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.390230 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.397113 2572 manager.go:216] Machine: {Timestamp:2026-07-08 19:42:09.395656521 +0000 UTC m=+0.560206915 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100791 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269d86a12d0f6a1619089fdba9fed4 SystemUUID:ec269d86-a12d-0f6a-1619-089fdba9fed4 BootID:c7c74bce-fbe5-4454-846f-4a793c8ba265 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:aa:bc:19:0d:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:aa:bc:19:0d:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:a4:9b:a2:ed:be Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.397262 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.397367 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.399703 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.399800 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-22.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.400831 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.400845 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.400859 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.401559 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.403052 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.406261 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.406406 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.406438 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.406451 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.406468 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.406477 2572 kubelet.go:398] "Adding apiserver pod source" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.406486 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 19:42:09.411495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.406842 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 19:42:09.412133 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.406856 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 19:42:09.412133 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.409321 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 19:42:09.412133 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.411535 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 19:42:09.412133 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.411574 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.412889 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.412906 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.412916 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.412924 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.412937 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.412944 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.412955 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.412963 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.412974 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.412982 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.413001 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 19:42:09.413444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.413009 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 19:42:09.414544 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.413675 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 19:42:09.414544 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.413686 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 19:42:09.418100 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.417690 2572 server.go:1268] "Started kubelet" Jul 08 19:42:09.420206 ip-10-0-131-22 systemd[1]: Started Kubernetes Kubelet. Jul 08 19:42:09.423214 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.422986 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 19:42:09.423214 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.423143 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 19:42:09.424233 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.423414 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.425973 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.426610 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:09.428477 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.429092 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.429490 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:09.431200 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-22.ec2.internal\" not found" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.432034 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.432048 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.432254 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.432325 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.432331 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.433331 2572 factory.go:55] Registering systemd factory Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.433347 2572 factory.go:223] Registration of the systemd container factory successfully Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.433571 2572 factory.go:153] Registering CRI-O factory Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.433582 2572 factory.go:223] Registration of the crio container factory successfully Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.433639 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.433658 2572 factory.go:103] Registering Raw factory Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.433668 2572 manager.go:1196] Started watching for new ooms in manager Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.433967 2572 manager.go:349] Starting recovery of all containers Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.442963 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.453028 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.455799 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.456037 2572 manager.go:354] Recovery completed Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.456312 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-131-22.ec2.internal" not found Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:09.456312 2572 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-22.ec2.internal\" not found" node="ip-10-0-131-22.ec2.internal" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.460656 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.465348 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-22.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.465380 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.465391 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-22.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.466005 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.466028 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.466053 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.466076 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.466969 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-22.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.466993 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.467003 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-22.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.467530 2572 policy_none.go:49] "None policy: Start" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.467550 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.467561 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.468646 2572 policy_none.go:47] "Start" Jul 08 19:42:09.507520 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.481348 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-131-22.ec2.internal" not found Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.525982 2572 manager.go:337] "Starting Device Plugin manager" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:09.526256 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.526269 2572 server.go:87] "Starting device plugin registration server" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.526562 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.526574 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.526659 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.526827 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.526838 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:09.527396 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:09.527441 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-22.ec2.internal\" not found" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.545031 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-131-22.ec2.internal" not found Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.627416 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.628442 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-22.ec2.internal" event="NodeHasSufficientMemory" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.628472 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-22.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.628485 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-22.ec2.internal" event="NodeHasSufficientPID" Jul 08 19:42:09.635424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.628513 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-22.ec2.internal" Jul 08 19:42:09.771027 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.638588 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-22.ec2.internal" Jul 08 19:42:09.771027 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:09.638617 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-22.ec2.internal\": node \"ip-10-0-131-22.ec2.internal\" not found" Jul 08 19:42:09.771027 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.665663 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 19:42:09.771027 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.666958 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 19:42:09.771027 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.667018 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 19:42:09.771027 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.667044 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 19:42:09.771027 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:09.667086 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 19:42:09.771027 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.669084 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:42:09.771027 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.767498 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal"] Jul 08 19:42:09.771027 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.770977 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" Jul 08 19:42:09.772977 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.772044 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal" Jul 08 19:42:09.898683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.817061 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" Jul 08 19:42:09.898683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.821997 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal" Jul 08 19:42:09.898683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.829421 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 19:42:09.898683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.832055 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 19:42:09.898683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.833462 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3c59695851aec142a5ca708972ab540-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal\" (UID: \"a3c59695851aec142a5ca708972ab540\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" Jul 08 19:42:09.898683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.833491 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c59695851aec142a5ca708972ab540-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal\" (UID: \"a3c59695851aec142a5ca708972ab540\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" Jul 08 19:42:09.898683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.833509 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ee3def2b49a5eb35498d1cdd8d47242-config\") pod \"kube-apiserver-proxy-ip-10-0-131-22.ec2.internal\" (UID: \"9ee3def2b49a5eb35498d1cdd8d47242\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal" Jul 08 19:42:10.044630 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.933963 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3c59695851aec142a5ca708972ab540-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal\" (UID: \"a3c59695851aec142a5ca708972ab540\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" Jul 08 19:42:10.044630 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.934002 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c59695851aec142a5ca708972ab540-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal\" (UID: \"a3c59695851aec142a5ca708972ab540\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" Jul 08 19:42:10.044630 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.934020 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ee3def2b49a5eb35498d1cdd8d47242-config\") pod \"kube-apiserver-proxy-ip-10-0-131-22.ec2.internal\" (UID: \"9ee3def2b49a5eb35498d1cdd8d47242\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal" Jul 08 19:42:10.044630 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.934055 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3c59695851aec142a5ca708972ab540-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal\" (UID: \"a3c59695851aec142a5ca708972ab540\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" Jul 08 19:42:10.044630 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.934064 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a3c59695851aec142a5ca708972ab540-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal\" (UID: \"a3c59695851aec142a5ca708972ab540\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" Jul 08 19:42:10.044630 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:09.934101 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9ee3def2b49a5eb35498d1cdd8d47242-config\") pod \"kube-apiserver-proxy-ip-10-0-131-22.ec2.internal\" (UID: \"9ee3def2b49a5eb35498d1cdd8d47242\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal" Jul 08 19:42:10.168675 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.133245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" Jul 08 19:42:10.168675 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.136274 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal" Jul 08 19:42:10.265799 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.217522 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 19:42:10.265799 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.217700 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 19:42:10.265799 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.217748 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 19:42:10.265799 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.217742 2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 19:42:10.289169 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.289104 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 19:37:09 +0000 UTC" deadline="2027-12-11 13:36:48.125433004 +0000 UTC" Jul 08 19:42:10.289169 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.289174 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12497h54m37.836260971s" Jul 08 19:42:10.407687 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.407651 2572 apiserver.go:52] "Watching apiserver" Jul 08 19:42:10.413770 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.413742 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 19:42:10.414957 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.414932 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-pr9w2","openshift-network-diagnostics/network-check-target-z946k","kube-system/global-pull-secret-syncer-q5dlc","openshift-cluster-node-tuning-operator/tuned-xdxgf","openshift-image-registry/node-ca-t7s4w","openshift-multus/multus-cwzwf","openshift-network-operator/iptables-alerter-8lvjh","openshift-ovn-kubernetes/ovnkube-node-tz7p9","kube-system/konnectivity-agent-v4rbh","kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk","openshift-dns/node-resolver-jx9bx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal","openshift-multus/multus-additional-cni-plugins-852tk"] Jul 08 19:42:10.416490 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.416467 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:10.416574 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.416546 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:10.417599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.417577 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:10.417666 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.417648 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:10.418595 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.418577 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:10.418666 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.418623 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:10.419727 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.419676 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.420739 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.420714 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.421537 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.421513 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:42:10.421637 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.421551 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 19:42:10.421637 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.421575 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-czlt8\"" Jul 08 19:42:10.421940 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.421926 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.422518 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.422500 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 19:42:10.422598 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.422528 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 19:42:10.422598 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.422572 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 19:42:10.422699 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.422667 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kdg82\"" Jul 08 19:42:10.423009 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.422996 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.423459 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.423443 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 19:42:10.423895 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.423882 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 19:42:10.423946 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.423926 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 19:42:10.423984 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.423967 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p6brb\"" Jul 08 19:42:10.424113 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.423980 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 19:42:10.424329 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.424316 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.424761 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.424742 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 19:42:10.424878 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.424863 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:42:10.425072 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.425058 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rg4h2\"" Jul 08 19:42:10.425142 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.425102 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 19:42:10.425607 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.425591 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:10.426413 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.426399 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 19:42:10.426580 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.426568 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 19:42:10.426642 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.426596 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 19:42:10.426817 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.426802 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 19:42:10.426875 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.426832 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.426875 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.426841 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 19:42:10.426875 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.426862 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bjl5c\"" Jul 08 19:42:10.427013 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.426920 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 19:42:10.427481 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.427467 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 19:42:10.427549 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.427539 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-grcl8\"" Jul 08 19:42:10.427688 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.427674 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 19:42:10.428405 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.428392 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.429018 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.429003 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b2pmd\"" Jul 08 19:42:10.429245 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.429227 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 19:42:10.429313 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.429265 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 19:42:10.429313 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.429236 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 19:42:10.429705 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.429691 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.430268 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.430253 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 19:42:10.430333 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.430261 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 19:42:10.430404 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.430378 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dxlr6\"" Jul 08 19:42:10.431232 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.431216 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 19:42:10.431432 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.431418 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v42gv\"" Jul 08 19:42:10.431568 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.431557 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 19:42:10.431619 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.431606 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 19:42:10.435827 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.434620 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 19:42:10.435827 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.435581 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.435827 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.435622 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4x2h\" (UniqueName: \"kubernetes.io/projected/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-kube-api-access-z4x2h\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.435827 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.435658 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-var-lib-cni-bin\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.435827 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.435689 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-etc-kubernetes\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.435827 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.435751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-os-release\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.435827 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.435799 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.436194 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.435851 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-socket-dir-parent\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.436194 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.435880 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-modprobe-d\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.436194 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.435927 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-lib-modules\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.436194 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.436016 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/392e7cb3-646e-4452-bf3c-0e44dae9bbff-cni-binary-copy\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.436194 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.436058 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g79l7\" (UniqueName: \"kubernetes.io/projected/392e7cb3-646e-4452-bf3c-0e44dae9bbff-kube-api-access-g79l7\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.436556 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.436120 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-hostroot\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.436874 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.436844 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53cf257b-7924-4126-9711-b432a663371a-hosts-file\") pod \"node-resolver-jx9bx\" (UID: \"53cf257b-7924-4126-9711-b432a663371a\") " pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.436955 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.436882 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-var-lib-cni-multus\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.436955 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.436900 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-var-lib-kubelet\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.437069 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.436961 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:10.437069 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.436982 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-var-lib-kubelet\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.437069 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437008 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-system-cni-dir\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.437069 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437032 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-slash\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.437069 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437049 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-var-lib-openvswitch\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.437069 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437063 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-registration-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.437269 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437096 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c05b3915-e70c-4244-a1f4-f7e22a7fd036-host-slash\") pod \"iptables-alerter-8lvjh\" (UID: \"c05b3915-e70c-4244-a1f4-f7e22a7fd036\") " pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.437269 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437134 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b75c8d82-d083-4cf2-9094-51739c38d0e5-host\") pod \"node-ca-t7s4w\" (UID: \"b75c8d82-d083-4cf2-9094-51739c38d0e5\") " pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.437269 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437195 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-run-netns\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.437269 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437226 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-conf-dir\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.437269 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437246 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-run-systemd\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.437269 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437263 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437280 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-socket-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437300 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-sysconfig\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437324 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-kubernetes\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437343 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-cni-dir\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-os-release\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437386 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83d6a6ae-387e-4c21-94af-9566defef5ca-cni-binary-copy\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437406 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-run-ovn\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437426 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-cni-netd\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437444 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437464 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:10.437517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437482 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-tuned\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437502 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437558 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trrtn\" (UniqueName: \"kubernetes.io/projected/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-kube-api-access-trrtn\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437582 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d39c7b30-0ef8-46ad-b195-277e67817f64-kubelet-config\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437599 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-sys\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437614 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c05b3915-e70c-4244-a1f4-f7e22a7fd036-iptables-alerter-script\") pod \"iptables-alerter-8lvjh\" (UID: \"c05b3915-e70c-4244-a1f4-f7e22a7fd036\") " pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437628 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/392e7cb3-646e-4452-bf3c-0e44dae9bbff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b75c8d82-d083-4cf2-9094-51739c38d0e5-serviceca\") pod \"node-ca-t7s4w\" (UID: \"b75c8d82-d083-4cf2-9094-51739c38d0e5\") " pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437687 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-run-multus-certs\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437711 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69a75894-7456-4c45-aa91-fa70553e36ca-ovnkube-config\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437734 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69a75894-7456-4c45-aa91-fa70553e36ca-ovnkube-script-lib\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437754 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-sysctl-d\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437768 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-tmp\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437783 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sck8t\" (UniqueName: \"kubernetes.io/projected/c05b3915-e70c-4244-a1f4-f7e22a7fd036-kube-api-access-sck8t\") pod \"iptables-alerter-8lvjh\" (UID: \"c05b3915-e70c-4244-a1f4-f7e22a7fd036\") " pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437799 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-daemon-config\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437821 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw74n\" (UniqueName: \"kubernetes.io/projected/83d6a6ae-387e-4c21-94af-9566defef5ca-kube-api-access-mw74n\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.437941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437840 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-systemd-units\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437854 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-node-log\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437872 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7g6k\" (UniqueName: \"kubernetes.io/projected/69a75894-7456-4c45-aa91-fa70553e36ca-kube-api-access-g7g6k\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.437889 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d39c7b30-0ef8-46ad-b195-277e67817f64-dbus\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438031 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-system-cni-dir\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438071 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69a75894-7456-4c45-aa91-fa70553e36ca-ovn-node-metrics-cert\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438103 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-sys-fs\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-sysctl-conf\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438147 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8131696-a0e8-4bda-a4c4-6ec7ea877e19-agent-certs\") pod \"konnectivity-agent-v4rbh\" (UID: \"f8131696-a0e8-4bda-a4c4-6ec7ea877e19\") " pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438189 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-systemd\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438211 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-host\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438237 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-cnibin\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438259 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-cnibin\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438282 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-run-openvswitch\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438306 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbwv9\" (UniqueName: \"kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9\") pod \"network-check-target-z946k\" (UID: \"0ca20598-adcb-4c45-87a1-91df3d8a8806\") " pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438329 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53cf257b-7924-4126-9711-b432a663371a-tmp-dir\") pod \"node-resolver-jx9bx\" (UID: \"53cf257b-7924-4126-9711-b432a663371a\") " pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438352 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89l2\" (UniqueName: \"kubernetes.io/projected/b75c8d82-d083-4cf2-9094-51739c38d0e5-kube-api-access-n89l2\") pod \"node-ca-t7s4w\" (UID: \"b75c8d82-d083-4cf2-9094-51739c38d0e5\") " pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.438666 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438375 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-kubelet\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438398 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-run-netns\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438422 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnktv\" (UniqueName: \"kubernetes.io/projected/cc99e320-6751-4034-9446-70884569cdbb-kube-api-access-fnktv\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438444 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ph9w\" (UniqueName: \"kubernetes.io/projected/53cf257b-7924-4126-9711-b432a663371a-kube-api-access-9ph9w\") pod \"node-resolver-jx9bx\" (UID: \"53cf257b-7924-4126-9711-b432a663371a\") " pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438470 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/392e7cb3-646e-4452-bf3c-0e44dae9bbff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438510 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-run-k8s-cni-cncf-io\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438533 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-etc-openvswitch\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438560 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-cni-bin\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438583 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69a75894-7456-4c45-aa91-fa70553e36ca-env-overrides\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438608 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8131696-a0e8-4bda-a4c4-6ec7ea877e19-konnectivity-ca\") pod \"konnectivity-agent-v4rbh\" (UID: \"f8131696-a0e8-4bda-a4c4-6ec7ea877e19\") " pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438632 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-device-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438654 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-run\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.438688 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-log-socket\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.439202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.439168 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 19:42:10.461634 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.461610 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-drwp4" Jul 08 19:42:10.475274 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.475248 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-drwp4" Jul 08 19:42:10.539839 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.539810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-kubernetes\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.539986 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.539847 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-cni-dir\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.539986 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.539862 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-os-release\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.539986 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.539882 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83d6a6ae-387e-4c21-94af-9566defef5ca-cni-binary-copy\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.539986 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.539939 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-kubernetes\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.539986 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.539962 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-run-ovn\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.539989 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-run-ovn\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.539963 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-os-release\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.539991 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-cni-netd\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540023 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540025 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-cni-dir\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540042 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540062 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-tuned\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540076 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540094 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-cni-netd\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trrtn\" (UniqueName: \"kubernetes.io/projected/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-kube-api-access-trrtn\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540096 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540124 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d39c7b30-0ef8-46ad-b195-277e67817f64-kubelet-config\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-sys\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c05b3915-e70c-4244-a1f4-f7e22a7fd036-iptables-alerter-script\") pod \"iptables-alerter-8lvjh\" (UID: \"c05b3915-e70c-4244-a1f4-f7e22a7fd036\") " pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.540183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540184 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-etc-selinux\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.540202 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540237 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-sys\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540235 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d39c7b30-0ef8-46ad-b195-277e67817f64-kubelet-config\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540206 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/392e7cb3-646e-4452-bf3c-0e44dae9bbff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.540313 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret podName:d39c7b30-0ef8-46ad-b195-277e67817f64 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:11.040267953 +0000 UTC m=+2.204818350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret") pod "global-pull-secret-syncer-q5dlc" (UID: "d39c7b30-0ef8-46ad-b195-277e67817f64") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540366 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b75c8d82-d083-4cf2-9094-51739c38d0e5-serviceca\") pod \"node-ca-t7s4w\" (UID: \"b75c8d82-d083-4cf2-9094-51739c38d0e5\") " pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540398 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-run-multus-certs\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69a75894-7456-4c45-aa91-fa70553e36ca-ovnkube-config\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540469 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69a75894-7456-4c45-aa91-fa70553e36ca-ovnkube-script-lib\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540510 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-sysctl-d\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540490 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540539 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83d6a6ae-387e-4c21-94af-9566defef5ca-cni-binary-copy\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-tmp\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540611 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sck8t\" (UniqueName: \"kubernetes.io/projected/c05b3915-e70c-4244-a1f4-f7e22a7fd036-kube-api-access-sck8t\") pod \"iptables-alerter-8lvjh\" (UID: \"c05b3915-e70c-4244-a1f4-f7e22a7fd036\") " pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540641 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-daemon-config\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540661 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-sysctl-d\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540666 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw74n\" (UniqueName: \"kubernetes.io/projected/83d6a6ae-387e-4c21-94af-9566defef5ca-kube-api-access-mw74n\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.540959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540695 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-systemd-units\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540720 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-node-log\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540755 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7g6k\" (UniqueName: \"kubernetes.io/projected/69a75894-7456-4c45-aa91-fa70553e36ca-kube-api-access-g7g6k\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540780 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d39c7b30-0ef8-46ad-b195-277e67817f64-dbus\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540805 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-system-cni-dir\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69a75894-7456-4c45-aa91-fa70553e36ca-ovn-node-metrics-cert\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540854 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-sys-fs\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540861 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b75c8d82-d083-4cf2-9094-51739c38d0e5-serviceca\") pod \"node-ca-t7s4w\" (UID: \"b75c8d82-d083-4cf2-9094-51739c38d0e5\") " pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540876 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-sysctl-conf\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540865 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c05b3915-e70c-4244-a1f4-f7e22a7fd036-iptables-alerter-script\") pod \"iptables-alerter-8lvjh\" (UID: \"c05b3915-e70c-4244-a1f4-f7e22a7fd036\") " pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540896 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/392e7cb3-646e-4452-bf3c-0e44dae9bbff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540901 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8131696-a0e8-4bda-a4c4-6ec7ea877e19-agent-certs\") pod \"konnectivity-agent-v4rbh\" (UID: \"f8131696-a0e8-4bda-a4c4-6ec7ea877e19\") " pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540928 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-systemd\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540951 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-host\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540956 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-node-log\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.540973 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-cnibin\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541043 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-run-multus-certs\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541110 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69a75894-7456-4c45-aa91-fa70553e36ca-ovnkube-config\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.541801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541177 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-cnibin\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541207 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-sys-fs\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541201 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-system-cni-dir\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541245 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-run-openvswitch\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541246 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-cnibin\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-run-openvswitch\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541278 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-host\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541288 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwv9\" (UniqueName: \"kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9\") pod \"network-check-target-z946k\" (UID: \"0ca20598-adcb-4c45-87a1-91df3d8a8806\") " pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541330 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-sysctl-conf\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541357 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69a75894-7456-4c45-aa91-fa70553e36ca-ovnkube-script-lib\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541365 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-systemd-units\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541572 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-systemd\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541588 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-daemon-config\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541625 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d39c7b30-0ef8-46ad-b195-277e67817f64-dbus\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541623 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53cf257b-7924-4126-9711-b432a663371a-tmp-dir\") pod \"node-resolver-jx9bx\" (UID: \"53cf257b-7924-4126-9711-b432a663371a\") " pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541680 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89l2\" (UniqueName: \"kubernetes.io/projected/b75c8d82-d083-4cf2-9094-51739c38d0e5-kube-api-access-n89l2\") pod \"node-ca-t7s4w\" (UID: \"b75c8d82-d083-4cf2-9094-51739c38d0e5\") " pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-kubelet\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541909 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-run-netns\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.542570 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541938 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnktv\" (UniqueName: \"kubernetes.io/projected/cc99e320-6751-4034-9446-70884569cdbb-kube-api-access-fnktv\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.541979 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ph9w\" (UniqueName: \"kubernetes.io/projected/53cf257b-7924-4126-9711-b432a663371a-kube-api-access-9ph9w\") pod \"node-resolver-jx9bx\" (UID: \"53cf257b-7924-4126-9711-b432a663371a\") " pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542009 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/392e7cb3-646e-4452-bf3c-0e44dae9bbff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542036 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-run-k8s-cni-cncf-io\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542062 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-etc-openvswitch\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542131 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-cni-bin\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542176 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69a75894-7456-4c45-aa91-fa70553e36ca-env-overrides\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542205 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8131696-a0e8-4bda-a4c4-6ec7ea877e19-konnectivity-ca\") pod \"konnectivity-agent-v4rbh\" (UID: \"f8131696-a0e8-4bda-a4c4-6ec7ea877e19\") " pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542230 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-device-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542253 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-run\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542278 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-log-socket\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542304 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542308 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-cnibin\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542331 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4x2h\" (UniqueName: \"kubernetes.io/projected/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-kube-api-access-z4x2h\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542408 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-cni-bin\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542409 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-device-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542445 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-etc-openvswitch\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.543456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542483 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-var-lib-cni-bin\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542510 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-etc-kubernetes\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542539 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-os-release\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542566 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-socket-dir-parent\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542614 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-modprobe-d\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542617 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53cf257b-7924-4126-9711-b432a663371a-tmp-dir\") pod \"node-resolver-jx9bx\" (UID: \"53cf257b-7924-4126-9711-b432a663371a\") " pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542638 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-lib-modules\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542621 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-run\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542662 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/392e7cb3-646e-4452-bf3c-0e44dae9bbff-cni-binary-copy\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542702 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-var-lib-cni-bin\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542812 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542850 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-run-netns\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542883 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-etc-kubernetes\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.542933 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-os-release\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.543478 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.543741 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-socket-dir-parent\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544192 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.543820 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-modprobe-d\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.543864 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-kubelet\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544006 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f8131696-a0e8-4bda-a4c4-6ec7ea877e19-konnectivity-ca\") pod \"konnectivity-agent-v4rbh\" (UID: \"f8131696-a0e8-4bda-a4c4-6ec7ea877e19\") " pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544098 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/392e7cb3-646e-4452-bf3c-0e44dae9bbff-cni-binary-copy\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544277 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-tmp\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544334 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-log-socket\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544393 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-run-k8s-cni-cncf-io\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544433 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-tuned\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544529 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-lib-modules\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g79l7\" (UniqueName: \"kubernetes.io/projected/392e7cb3-646e-4452-bf3c-0e44dae9bbff-kube-api-access-g79l7\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-hostroot\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53cf257b-7924-4126-9711-b432a663371a-hosts-file\") pod \"node-resolver-jx9bx\" (UID: \"53cf257b-7924-4126-9711-b432a663371a\") " pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544747 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-var-lib-cni-multus\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-var-lib-kubelet\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544863 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-var-lib-kubelet\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544849 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/392e7cb3-646e-4452-bf3c-0e44dae9bbff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544910 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f8131696-a0e8-4bda-a4c4-6ec7ea877e19-agent-certs\") pod \"konnectivity-agent-v4rbh\" (UID: \"f8131696-a0e8-4bda-a4c4-6ec7ea877e19\") " pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:10.544973 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544944 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53cf257b-7924-4126-9711-b432a663371a-hosts-file\") pod \"node-resolver-jx9bx\" (UID: \"53cf257b-7924-4126-9711-b432a663371a\") " pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.544987 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-var-lib-cni-multus\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545041 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-hostroot\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545085 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545124 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-var-lib-kubelet\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545218 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-system-cni-dir\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545258 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-slash\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545291 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-var-lib-openvswitch\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545296 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-var-lib-kubelet\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545325 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-registration-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545353 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c05b3915-e70c-4244-a1f4-f7e22a7fd036-host-slash\") pod \"iptables-alerter-8lvjh\" (UID: \"c05b3915-e70c-4244-a1f4-f7e22a7fd036\") " pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545385 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b75c8d82-d083-4cf2-9094-51739c38d0e5-host\") pod \"node-ca-t7s4w\" (UID: \"b75c8d82-d083-4cf2-9094-51739c38d0e5\") " pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545419 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-run-netns\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545470 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-conf-dir\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545502 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-run-systemd\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545530 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545563 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-socket-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.545825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545613 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-sysconfig\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545629 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-var-lib-openvswitch\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545644 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69a75894-7456-4c45-aa91-fa70553e36ca-env-overrides\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545711 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b75c8d82-d083-4cf2-9094-51739c38d0e5-host\") pod \"node-ca-t7s4w\" (UID: \"b75c8d82-d083-4cf2-9094-51739c38d0e5\") " pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545794 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-host-run-netns\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545806 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-socket-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545562 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69a75894-7456-4c45-aa91-fa70553e36ca-ovn-node-metrics-cert\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545859 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-etc-sysconfig\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545896 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c05b3915-e70c-4244-a1f4-f7e22a7fd036-host-slash\") pod \"iptables-alerter-8lvjh\" (UID: \"c05b3915-e70c-4244-a1f4-f7e22a7fd036\") " pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545924 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-slash\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545946 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/392e7cb3-646e-4452-bf3c-0e44dae9bbff-system-cni-dir\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545949 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-registration-dir\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545899 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-run-systemd\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545971 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83d6a6ae-387e-4c21-94af-9566defef5ca-multus-conf-dir\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.546599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.545979 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69a75894-7456-4c45-aa91-fa70553e36ca-host-run-ovn-kubernetes\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.547886 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.547862 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:10.547975 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.547940 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs podName:cc99e320-6751-4034-9446-70884569cdbb nodeName:}" failed. No retries permitted until 2026-07-08 19:42:11.047921691 +0000 UTC m=+2.212472079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs") pod "network-metrics-daemon-pr9w2" (UID: "cc99e320-6751-4034-9446-70884569cdbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:10.549543 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.549519 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:42:10.549543 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.549545 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:42:10.550434 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.549560 2572 projected.go:196] Error preparing data for projected volume kube-api-access-pbwv9 for pod openshift-network-diagnostics/network-check-target-z946k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:10.550434 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:10.549638 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9 podName:0ca20598-adcb-4c45-87a1-91df3d8a8806 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:11.049620017 +0000 UTC m=+2.214170418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pbwv9" (UniqueName: "kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9") pod "network-check-target-z946k" (UID: "0ca20598-adcb-4c45-87a1-91df3d8a8806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:10.552332 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.552196 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sck8t\" (UniqueName: \"kubernetes.io/projected/c05b3915-e70c-4244-a1f4-f7e22a7fd036-kube-api-access-sck8t\") pod \"iptables-alerter-8lvjh\" (UID: \"c05b3915-e70c-4244-a1f4-f7e22a7fd036\") " pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.552332 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.552251 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ph9w\" (UniqueName: \"kubernetes.io/projected/53cf257b-7924-4126-9711-b432a663371a-kube-api-access-9ph9w\") pod \"node-resolver-jx9bx\" (UID: \"53cf257b-7924-4126-9711-b432a663371a\") " pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.552674 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.552645 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7g6k\" (UniqueName: \"kubernetes.io/projected/69a75894-7456-4c45-aa91-fa70553e36ca-kube-api-access-g7g6k\") pod \"ovnkube-node-tz7p9\" (UID: \"69a75894-7456-4c45-aa91-fa70553e36ca\") " pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.553127 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.553099 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89l2\" (UniqueName: \"kubernetes.io/projected/b75c8d82-d083-4cf2-9094-51739c38d0e5-kube-api-access-n89l2\") pod \"node-ca-t7s4w\" (UID: \"b75c8d82-d083-4cf2-9094-51739c38d0e5\") " pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.553243 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.553199 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trrtn\" (UniqueName: \"kubernetes.io/projected/3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337-kube-api-access-trrtn\") pod \"aws-ebs-csi-driver-node-pgxdk\" (UID: \"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.553315 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.553293 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw74n\" (UniqueName: \"kubernetes.io/projected/83d6a6ae-387e-4c21-94af-9566defef5ca-kube-api-access-mw74n\") pod \"multus-cwzwf\" (UID: \"83d6a6ae-387e-4c21-94af-9566defef5ca\") " pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.553655 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.553638 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4x2h\" (UniqueName: \"kubernetes.io/projected/ab4bfcd0-897f-42b2-af5c-c63faf16e27e-kube-api-access-z4x2h\") pod \"tuned-xdxgf\" (UID: \"ab4bfcd0-897f-42b2-af5c-c63faf16e27e\") " pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.553879 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.553859 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnktv\" (UniqueName: \"kubernetes.io/projected/cc99e320-6751-4034-9446-70884569cdbb-kube-api-access-fnktv\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:10.554532 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.554513 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g79l7\" (UniqueName: \"kubernetes.io/projected/392e7cb3-646e-4452-bf3c-0e44dae9bbff-kube-api-access-g79l7\") pod \"multus-additional-cni-plugins-852tk\" (UID: \"392e7cb3-646e-4452-bf3c-0e44dae9bbff\") " pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.567550 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.567531 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-852tk" Jul 08 19:42:10.714941 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:10.714899 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c59695851aec142a5ca708972ab540.slice/crio-90251183e98db3d43e34c7ec58b2ef36620777eb329483256d3c50dfb7fecfca WatchSource:0}: Error finding container 90251183e98db3d43e34c7ec58b2ef36620777eb329483256d3c50dfb7fecfca: Status 404 returned error can't find the container with id 90251183e98db3d43e34c7ec58b2ef36620777eb329483256d3c50dfb7fecfca Jul 08 19:42:10.715907 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:10.715885 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee3def2b49a5eb35498d1cdd8d47242.slice/crio-4ab4f0067f4cca9ca24ec7b3241d5e372f910c222e525b7e4ec9164825f12025 WatchSource:0}: Error finding container 4ab4f0067f4cca9ca24ec7b3241d5e372f910c222e525b7e4ec9164825f12025: Status 404 returned error can't find the container with id 4ab4f0067f4cca9ca24ec7b3241d5e372f910c222e525b7e4ec9164825f12025 Jul 08 19:42:10.719418 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.719403 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:42:10.752586 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.752540 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" Jul 08 19:42:10.758595 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.758577 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t7s4w" Jul 08 19:42:10.761023 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:10.760993 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4bfcd0_897f_42b2_af5c_c63faf16e27e.slice/crio-09d28e390f75adba98752395ec53c098da898f6512a7743b6638d8956a5ea3a2 WatchSource:0}: Error finding container 09d28e390f75adba98752395ec53c098da898f6512a7743b6638d8956a5ea3a2: Status 404 returned error can't find the container with id 09d28e390f75adba98752395ec53c098da898f6512a7743b6638d8956a5ea3a2 Jul 08 19:42:10.766704 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:10.766680 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75c8d82_d083_4cf2_9094_51739c38d0e5.slice/crio-03c48502111e8c12c30fabf85b42390b78db313cad4abc8b7e7930e2e37691e0 WatchSource:0}: Error finding container 03c48502111e8c12c30fabf85b42390b78db313cad4abc8b7e7930e2e37691e0: Status 404 returned error can't find the container with id 03c48502111e8c12c30fabf85b42390b78db313cad4abc8b7e7930e2e37691e0 Jul 08 19:42:10.775591 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.775566 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cwzwf" Jul 08 19:42:10.783921 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:10.783896 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d6a6ae_387e_4c21_94af_9566defef5ca.slice/crio-a9573d7fd8fa25621367b2324be4d4028aa73a32e3c4ea8d60985f0a6f481851 WatchSource:0}: Error finding container a9573d7fd8fa25621367b2324be4d4028aa73a32e3c4ea8d60985f0a6f481851: Status 404 returned error can't find the container with id a9573d7fd8fa25621367b2324be4d4028aa73a32e3c4ea8d60985f0a6f481851 Jul 08 19:42:10.794759 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.794730 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8lvjh" Jul 08 19:42:10.802560 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:10.802520 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc05b3915_e70c_4244_a1f4_f7e22a7fd036.slice/crio-43ecf9171d0b303dedbca47831f8d68d443e6a94f0e598b119ed62972d1b2d74 WatchSource:0}: Error finding container 43ecf9171d0b303dedbca47831f8d68d443e6a94f0e598b119ed62972d1b2d74: Status 404 returned error can't find the container with id 43ecf9171d0b303dedbca47831f8d68d443e6a94f0e598b119ed62972d1b2d74 Jul 08 19:42:10.809887 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.809860 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:10.814955 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.814939 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:10.817301 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:10.817279 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a75894_7456_4c45_aa91_fa70553e36ca.slice/crio-ff59baa0a459fb4d0b18ff1b5596691be9cb82d819d80f30f13707836f3b7ac3 WatchSource:0}: Error finding container ff59baa0a459fb4d0b18ff1b5596691be9cb82d819d80f30f13707836f3b7ac3: Status 404 returned error can't find the container with id ff59baa0a459fb4d0b18ff1b5596691be9cb82d819d80f30f13707836f3b7ac3 Jul 08 19:42:10.823458 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:10.823433 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8131696_a0e8_4bda_a4c4_6ec7ea877e19.slice/crio-4e37ea2b41c11600a1c2c99b252d2455e4f7dba4db01424b889cbcddc9b0b67b WatchSource:0}: Error finding container 4e37ea2b41c11600a1c2c99b252d2455e4f7dba4db01424b889cbcddc9b0b67b: Status 404 returned error can't find the container with id 4e37ea2b41c11600a1c2c99b252d2455e4f7dba4db01424b889cbcddc9b0b67b Jul 08 19:42:10.828191 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.828171 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" Jul 08 19:42:10.833904 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:10.833882 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jx9bx" Jul 08 19:42:10.836562 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:10.836536 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3217a1bd_e7aa_4004_a2cb_3f6b3fa6f337.slice/crio-3e1422e853e261db476d621cecfff1b548cb147beb5ff039bf0f983abc9614ce WatchSource:0}: Error finding container 3e1422e853e261db476d621cecfff1b548cb147beb5ff039bf0f983abc9614ce: Status 404 returned error can't find the container with id 3e1422e853e261db476d621cecfff1b548cb147beb5ff039bf0f983abc9614ce Jul 08 19:42:10.841303 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:10.841279 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53cf257b_7924_4126_9711_b432a663371a.slice/crio-e0bff1b31acc1a4e709dfc93b2834571c9d0276fd77ff052f6a365b8b8797b6d WatchSource:0}: Error finding container e0bff1b31acc1a4e709dfc93b2834571c9d0276fd77ff052f6a365b8b8797b6d: Status 404 returned error can't find the container with id e0bff1b31acc1a4e709dfc93b2834571c9d0276fd77ff052f6a365b8b8797b6d Jul 08 19:42:11.049878 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.049835 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwv9\" (UniqueName: \"kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9\") pod \"network-check-target-z946k\" (UID: \"0ca20598-adcb-4c45-87a1-91df3d8a8806\") " pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:11.049878 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.049887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:11.050108 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.049910 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:11.050108 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.049996 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:11.050108 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.050003 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:42:11.050108 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.050023 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:42:11.050108 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.050034 2572 projected.go:196] Error preparing data for projected volume kube-api-access-pbwv9 for pod openshift-network-diagnostics/network-check-target-z946k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:11.050108 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.050037 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:11.050108 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.050059 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret podName:d39c7b30-0ef8-46ad-b195-277e67817f64 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:12.050043012 +0000 UTC m=+3.214593398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret") pod "global-pull-secret-syncer-q5dlc" (UID: "d39c7b30-0ef8-46ad-b195-277e67817f64") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:11.050108 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.050078 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9 podName:0ca20598-adcb-4c45-87a1-91df3d8a8806 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:12.050064654 +0000 UTC m=+3.214615060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pbwv9" (UniqueName: "kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9") pod "network-check-target-z946k" (UID: "0ca20598-adcb-4c45-87a1-91df3d8a8806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:11.050108 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.050093 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs podName:cc99e320-6751-4034-9446-70884569cdbb nodeName:}" failed. No retries permitted until 2026-07-08 19:42:12.050087037 +0000 UTC m=+3.214637418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs") pod "network-metrics-daemon-pr9w2" (UID: "cc99e320-6751-4034-9446-70884569cdbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:11.319006 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.318926 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:42:11.477390 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.477346 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 19:37:10 +0000 UTC" deadline="2028-01-14 03:36:14.666731631 +0000 UTC" Jul 08 19:42:11.477390 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.477389 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13303h54m3.189348026s" Jul 08 19:42:11.674384 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.674330 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:11.674568 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.674527 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:11.675072 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.675047 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:11.675237 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.675213 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:11.675337 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.675324 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:11.675452 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:11.675434 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:11.683026 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.682906 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:42:11.698302 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.698210 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8lvjh" event={"ID":"c05b3915-e70c-4244-a1f4-f7e22a7fd036","Type":"ContainerStarted","Data":"43ecf9171d0b303dedbca47831f8d68d443e6a94f0e598b119ed62972d1b2d74"} Jul 08 19:42:11.722628 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.722493 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cwzwf" event={"ID":"83d6a6ae-387e-4c21-94af-9566defef5ca","Type":"ContainerStarted","Data":"a9573d7fd8fa25621367b2324be4d4028aa73a32e3c4ea8d60985f0a6f481851"} Jul 08 19:42:11.737326 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.737046 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 19:42:11.756866 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.756742 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v4rbh" event={"ID":"f8131696-a0e8-4bda-a4c4-6ec7ea877e19","Type":"ContainerStarted","Data":"4e37ea2b41c11600a1c2c99b252d2455e4f7dba4db01424b889cbcddc9b0b67b"} Jul 08 19:42:11.766598 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.766414 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" event={"ID":"69a75894-7456-4c45-aa91-fa70553e36ca","Type":"ContainerStarted","Data":"ff59baa0a459fb4d0b18ff1b5596691be9cb82d819d80f30f13707836f3b7ac3"} Jul 08 19:42:11.792765 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.792668 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t7s4w" event={"ID":"b75c8d82-d083-4cf2-9094-51739c38d0e5","Type":"ContainerStarted","Data":"03c48502111e8c12c30fabf85b42390b78db313cad4abc8b7e7930e2e37691e0"} Jul 08 19:42:11.837934 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.837569 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" event={"ID":"ab4bfcd0-897f-42b2-af5c-c63faf16e27e","Type":"ContainerStarted","Data":"09d28e390f75adba98752395ec53c098da898f6512a7743b6638d8956a5ea3a2"} Jul 08 19:42:11.872619 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.872579 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" event={"ID":"a3c59695851aec142a5ca708972ab540","Type":"ContainerStarted","Data":"90251183e98db3d43e34c7ec58b2ef36620777eb329483256d3c50dfb7fecfca"} Jul 08 19:42:11.881710 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.881666 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal" event={"ID":"9ee3def2b49a5eb35498d1cdd8d47242","Type":"ContainerStarted","Data":"4ab4f0067f4cca9ca24ec7b3241d5e372f910c222e525b7e4ec9164825f12025"} Jul 08 19:42:11.901821 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.901777 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-852tk" event={"ID":"392e7cb3-646e-4452-bf3c-0e44dae9bbff","Type":"ContainerStarted","Data":"55d9c4ab792eb65fe5030428f54a1eccb33a89400a26b23456aa232c8c49702d"} Jul 08 19:42:11.906319 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.906279 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jx9bx" event={"ID":"53cf257b-7924-4126-9711-b432a663371a","Type":"ContainerStarted","Data":"e0bff1b31acc1a4e709dfc93b2834571c9d0276fd77ff052f6a365b8b8797b6d"} Jul 08 19:42:11.946000 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:11.945959 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" event={"ID":"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337","Type":"ContainerStarted","Data":"3e1422e853e261db476d621cecfff1b548cb147beb5ff039bf0f983abc9614ce"} Jul 08 19:42:12.057896 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:12.057834 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwv9\" (UniqueName: \"kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9\") pod \"network-check-target-z946k\" (UID: \"0ca20598-adcb-4c45-87a1-91df3d8a8806\") " pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:12.057896 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:12.057898 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:12.058149 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:12.057941 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:12.058149 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:12.058076 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:12.058149 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:12.058143 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret podName:d39c7b30-0ef8-46ad-b195-277e67817f64 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:14.058123106 +0000 UTC m=+5.222673511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret") pod "global-pull-secret-syncer-q5dlc" (UID: "d39c7b30-0ef8-46ad-b195-277e67817f64") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:12.058733 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:12.058597 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:42:12.058733 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:12.058622 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:42:12.058733 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:12.058634 2572 projected.go:196] Error preparing data for projected volume kube-api-access-pbwv9 for pod openshift-network-diagnostics/network-check-target-z946k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:12.058733 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:12.058683 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9 podName:0ca20598-adcb-4c45-87a1-91df3d8a8806 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:14.0586676 +0000 UTC m=+5.223217996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pbwv9" (UniqueName: "kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9") pod "network-check-target-z946k" (UID: "0ca20598-adcb-4c45-87a1-91df3d8a8806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:12.058988 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:12.058823 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:12.058988 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:12.058865 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs podName:cc99e320-6751-4034-9446-70884569cdbb nodeName:}" failed. No retries permitted until 2026-07-08 19:42:14.058850424 +0000 UTC m=+5.223400819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs") pod "network-metrics-daemon-pr9w2" (UID: "cc99e320-6751-4034-9446-70884569cdbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:12.477970 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:12.477878 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 19:37:10 +0000 UTC" deadline="2028-01-30 13:58:30.34117638 +0000 UTC" Jul 08 19:42:12.477970 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:12.477937 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13698h16m17.863244027s" Jul 08 19:42:13.669210 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:13.667705 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:13.669210 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:13.667836 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:13.669210 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:13.668513 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:13.669210 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:13.668623 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:13.669210 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:13.668673 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:13.669210 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:13.668746 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:14.076667 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:14.076736 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:14.076782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwv9\" (UniqueName: \"kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9\") pod \"network-check-target-z946k\" (UID: \"0ca20598-adcb-4c45-87a1-91df3d8a8806\") " pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:14.076941 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:14.076963 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:14.076976 2572 projected.go:196] Error preparing data for projected volume kube-api-access-pbwv9 for pod openshift-network-diagnostics/network-check-target-z946k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:14.077038 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9 podName:0ca20598-adcb-4c45-87a1-91df3d8a8806 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:18.077018805 +0000 UTC m=+9.241569206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pbwv9" (UniqueName: "kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9") pod "network-check-target-z946k" (UID: "0ca20598-adcb-4c45-87a1-91df3d8a8806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:14.077512 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:14.077567 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs podName:cc99e320-6751-4034-9446-70884569cdbb nodeName:}" failed. No retries permitted until 2026-07-08 19:42:18.077551855 +0000 UTC m=+9.242102242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs") pod "network-metrics-daemon-pr9w2" (UID: "cc99e320-6751-4034-9446-70884569cdbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:14.077624 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:14.077806 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:14.077654 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret podName:d39c7b30-0ef8-46ad-b195-277e67817f64 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:18.07764469 +0000 UTC m=+9.242195071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret") pod "global-pull-secret-syncer-q5dlc" (UID: "d39c7b30-0ef8-46ad-b195-277e67817f64") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:15.672186 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:15.671713 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:15.672186 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:15.671836 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:15.672809 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:15.672754 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:15.672885 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:15.672754 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:15.672949 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:15.672875 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:15.673006 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:15.672957 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:17.667694 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:17.667659 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:17.668198 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:17.667799 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:17.668198 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:17.667818 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:17.668198 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:17.667926 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:17.668198 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:17.667959 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:17.668198 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:17.668034 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:18.117033 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:18.116973 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwv9\" (UniqueName: \"kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9\") pod \"network-check-target-z946k\" (UID: \"0ca20598-adcb-4c45-87a1-91df3d8a8806\") " pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:18.117228 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:18.117055 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:18.117228 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:18.117097 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:18.117357 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:18.117258 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:18.117357 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:18.117326 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret podName:d39c7b30-0ef8-46ad-b195-277e67817f64 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:26.117305945 +0000 UTC m=+17.281856343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret") pod "global-pull-secret-syncer-q5dlc" (UID: "d39c7b30-0ef8-46ad-b195-277e67817f64") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:18.117606 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:18.117582 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:18.117733 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:18.117631 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:42:18.117733 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:18.117657 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:42:18.117733 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:18.117671 2572 projected.go:196] Error preparing data for projected volume kube-api-access-pbwv9 for pod openshift-network-diagnostics/network-check-target-z946k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:18.117733 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:18.117636 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs podName:cc99e320-6751-4034-9446-70884569cdbb nodeName:}" failed. No retries permitted until 2026-07-08 19:42:26.117621799 +0000 UTC m=+17.282172183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs") pod "network-metrics-daemon-pr9w2" (UID: "cc99e320-6751-4034-9446-70884569cdbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:18.117733 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:18.117733 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9 podName:0ca20598-adcb-4c45-87a1-91df3d8a8806 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:26.117718091 +0000 UTC m=+17.282268477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pbwv9" (UniqueName: "kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9") pod "network-check-target-z946k" (UID: "0ca20598-adcb-4c45-87a1-91df3d8a8806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:19.669037 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:19.668991 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:19.669594 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:19.669149 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:19.669649 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:19.669628 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:19.669818 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:19.669763 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:19.670052 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:19.670027 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:19.670172 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:19.670138 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:21.667909 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:21.667805 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:21.667909 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:21.667849 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:21.668435 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:21.667954 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:21.668435 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:21.667805 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:21.668562 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:21.668446 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:21.668562 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:21.668536 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:23.668174 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:23.668134 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:23.668648 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:23.668134 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:23.668648 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:23.668285 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:23.668648 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:23.668397 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:23.668648 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:23.668133 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:23.668648 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:23.668560 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:25.667871 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:25.667834 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:25.667871 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:25.667857 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:25.668389 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:25.667840 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:25.668389 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:25.667966 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:25.668389 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:25.668121 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:25.668389 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:25.668241 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:26.179362 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:26.179319 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:26.179436 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:26.179373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwv9\" (UniqueName: \"kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9\") pod \"network-check-target-z946k\" (UID: \"0ca20598-adcb-4c45-87a1-91df3d8a8806\") " pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:26.179436 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:26.179408 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:26.179526 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:26.179494 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:26.179526 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:26.179517 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:26.179623 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:26.179541 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:42:26.179623 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:26.179564 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:42:26.179623 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:26.179570 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret podName:d39c7b30-0ef8-46ad-b195-277e67817f64 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:42.179548257 +0000 UTC m=+33.344098660 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret") pod "global-pull-secret-syncer-q5dlc" (UID: "d39c7b30-0ef8-46ad-b195-277e67817f64") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:26.179623 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:26.179578 2572 projected.go:196] Error preparing data for projected volume kube-api-access-pbwv9 for pod openshift-network-diagnostics/network-check-target-z946k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:26.179623 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:26.179591 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs podName:cc99e320-6751-4034-9446-70884569cdbb nodeName:}" failed. No retries permitted until 2026-07-08 19:42:42.179580068 +0000 UTC m=+33.344130449 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs") pod "network-metrics-daemon-pr9w2" (UID: "cc99e320-6751-4034-9446-70884569cdbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:26.179623 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:26.179623 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9 podName:0ca20598-adcb-4c45-87a1-91df3d8a8806 nodeName:}" failed. No retries permitted until 2026-07-08 19:42:42.179609071 +0000 UTC m=+33.344159474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pbwv9" (UniqueName: "kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9") pod "network-check-target-z946k" (UID: "0ca20598-adcb-4c45-87a1-91df3d8a8806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:27.668337 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:27.668304 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:27.668865 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:27.668305 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:27.668865 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:27.668445 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:27.668865 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:27.668477 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:27.668865 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:27.668601 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:27.668865 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:27.668699 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:29.671357 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:29.669496 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:29.671357 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:29.669888 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:29.671357 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:29.670398 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:29.671357 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:29.670529 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:29.671357 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:29.670587 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:29.671357 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:29.670688 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:30.988840 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:30.988792 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" event={"ID":"ab4bfcd0-897f-42b2-af5c-c63faf16e27e","Type":"ContainerStarted","Data":"e1eba96efede793d08a5eae3b59e0f2420f97d27a1270df7d9ae4f1a109ce012"} Jul 08 19:42:30.992437 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:30.992191 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal" event={"ID":"9ee3def2b49a5eb35498d1cdd8d47242","Type":"ContainerStarted","Data":"40c15a2d96ad14537f68cd2a0a319ed69c1e9619d157369107256380537b5159"} Jul 08 19:42:31.005296 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:31.005079 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xdxgf" podStartSLOduration=2.004268931 podStartE2EDuration="22.005054483s" totalImagesPullingTime="20.000785552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:31.004664301 +0000 UTC m=+22.169214718" watchObservedRunningTime="2026-07-08 19:42:31.005054483 +0000 UTC m=+22.169604887" Jul 08 19:42:31.016553 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:31.016446 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-22.ec2.internal" podStartSLOduration=22.016426213 podStartE2EDuration="22.016426213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:31.01628851 +0000 UTC m=+22.180838931" watchObservedRunningTime="2026-07-08 19:42:31.016426213 +0000 UTC m=+22.180976617" Jul 08 19:42:31.667651 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:31.667399 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:31.667867 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:31.667418 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:31.667867 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:31.667691 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:31.667867 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:31.667460 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:31.667974 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:31.667781 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:31.667974 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:31.667893 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:31.996633 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:31.996529 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cwzwf" event={"ID":"83d6a6ae-387e-4c21-94af-9566defef5ca","Type":"ContainerStarted","Data":"9ce8ddcfe9a64cc2605df89f99b1ec8a8a1809ceee3a0f8a21f06af7cfc1739a"} Jul 08 19:42:31.998306 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:31.998273 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v4rbh" event={"ID":"f8131696-a0e8-4bda-a4c4-6ec7ea877e19","Type":"ContainerStarted","Data":"e707aa9afc0206e4148aa6ad793e237f62aa3aec9580fb1c8ca247620f034b0f"} Jul 08 19:42:32.002108 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.002080 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" event={"ID":"69a75894-7456-4c45-aa91-fa70553e36ca","Type":"ContainerStarted","Data":"13e335bdf9764e8952229ae7d3b0df19a9818d0c24a12ad2af22d669c2fe778c"} Jul 08 19:42:32.002273 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.002150 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" event={"ID":"69a75894-7456-4c45-aa91-fa70553e36ca","Type":"ContainerStarted","Data":"c943732d7f4a378d54b8e9fbfc7f083e7b2787a69f0ea1633b8727766beaa881"} Jul 08 19:42:32.002273 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.002203 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" event={"ID":"69a75894-7456-4c45-aa91-fa70553e36ca","Type":"ContainerStarted","Data":"5d3a05b0b482acfbd9b8cb5637faf941c19667411fbd5ba86a787a16837ae0b7"} Jul 08 19:42:32.002273 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.002241 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" event={"ID":"69a75894-7456-4c45-aa91-fa70553e36ca","Type":"ContainerStarted","Data":"7a6b4707ae52d806907358916238db50df94e112dcd9577bf0e721e46bd3ce4f"} Jul 08 19:42:32.002273 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.002270 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" event={"ID":"69a75894-7456-4c45-aa91-fa70553e36ca","Type":"ContainerStarted","Data":"10837f206f7ed43ec95466f7e530442a228d22e8f939fecc8ffd8ff350bb7cfb"} Jul 08 19:42:32.002489 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.002307 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" event={"ID":"69a75894-7456-4c45-aa91-fa70553e36ca","Type":"ContainerStarted","Data":"90f380874b348454dc4a7a49fb7e6de3ee92d9673e20ff7c6c3ecbad67f8711f"} Jul 08 19:42:32.003814 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.003789 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t7s4w" event={"ID":"b75c8d82-d083-4cf2-9094-51739c38d0e5","Type":"ContainerStarted","Data":"6792da6b8366040a3c09063148b2a624793789e505d09da93e3bb99189d23fbd"} Jul 08 19:42:32.005374 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.005343 2572 generic.go:356] "Generic (PLEG): container finished" podID="a3c59695851aec142a5ca708972ab540" containerID="6239c79f1b9a098afbaa4cfa38fe9f21c53863c6c4639b9d73445fca961aa930" exitCode=0 Jul 08 19:42:32.005476 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.005406 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" event={"ID":"a3c59695851aec142a5ca708972ab540","Type":"ContainerDied","Data":"6239c79f1b9a098afbaa4cfa38fe9f21c53863c6c4639b9d73445fca961aa930"} Jul 08 19:42:32.007122 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.007102 2572 generic.go:356] "Generic (PLEG): container finished" podID="392e7cb3-646e-4452-bf3c-0e44dae9bbff" containerID="758ad47eb31361dcfab83f9820736bcfc5a4f69404df89fd311c013560f0133e" exitCode=0 Jul 08 19:42:32.007231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.007194 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-852tk" event={"ID":"392e7cb3-646e-4452-bf3c-0e44dae9bbff","Type":"ContainerDied","Data":"758ad47eb31361dcfab83f9820736bcfc5a4f69404df89fd311c013560f0133e"} Jul 08 19:42:32.008930 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.008902 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jx9bx" event={"ID":"53cf257b-7924-4126-9711-b432a663371a","Type":"ContainerStarted","Data":"6cf91d115e5641e60a434020fa5f282cfa0079db5a17732176d7348310f93101"} Jul 08 19:42:32.010630 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.010579 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" event={"ID":"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337","Type":"ContainerStarted","Data":"1ac6eeae6ff0e349848acebd8e65895918afce2e5849bbfc1654f8a3aa526865"} Jul 08 19:42:32.012518 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.012481 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cwzwf" podStartSLOduration=2.94653975 podStartE2EDuration="23.012464823s" totalImagesPullingTime="20.065925073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:32.012051694 +0000 UTC m=+23.176602098" watchObservedRunningTime="2026-07-08 19:42:32.012464823 +0000 UTC m=+23.177015238" Jul 08 19:42:32.026216 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.026133 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v4rbh" podStartSLOduration=3.046150221 podStartE2EDuration="23.026112463s" totalImagesPullingTime="19.979962242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:32.025370096 +0000 UTC m=+23.189920500" watchObservedRunningTime="2026-07-08 19:42:32.026112463 +0000 UTC m=+23.190662867" Jul 08 19:42:32.052461 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.052111 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jx9bx" podStartSLOduration=3.0859398 podStartE2EDuration="23.052088976s" totalImagesPullingTime="19.966149176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:32.051756931 +0000 UTC m=+23.216307334" watchObservedRunningTime="2026-07-08 19:42:32.052088976 +0000 UTC m=+23.216639379" Jul 08 19:42:32.094059 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.094007 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t7s4w" podStartSLOduration=3.056509009 podStartE2EDuration="23.093990671s" totalImagesPullingTime="20.037481662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:32.093072347 +0000 UTC m=+23.257622750" watchObservedRunningTime="2026-07-08 19:42:32.093990671 +0000 UTC m=+23.258541098" Jul 08 19:42:32.794726 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:32.794682 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 19:42:33.015913 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.015818 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" event={"ID":"a3c59695851aec142a5ca708972ab540","Type":"ContainerStarted","Data":"e889773feeda174c68b2ef4433d1ec4c3f2a760b40d10c9e0467d050f06f1003"} Jul 08 19:42:33.017893 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.017865 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" event={"ID":"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337","Type":"ContainerStarted","Data":"2963b48c847712eea48f4ef66b3183431bf289215b3818db06c65e3230bf723a"} Jul 08 19:42:33.019533 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.019503 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8lvjh" event={"ID":"c05b3915-e70c-4244-a1f4-f7e22a7fd036","Type":"ContainerStarted","Data":"6bb0fb5befc235bf8605c8938fec0f1774dcaaf457e367343fbbd6513ce88c4e"} Jul 08 19:42:33.032812 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.032755 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-22.ec2.internal" podStartSLOduration=24.03273819 podStartE2EDuration="24.03273819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:33.032210618 +0000 UTC m=+24.196761022" watchObservedRunningTime="2026-07-08 19:42:33.03273819 +0000 UTC m=+24.197288586" Jul 08 19:42:33.044758 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.044685 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8lvjh" podStartSLOduration=4.085282516 podStartE2EDuration="24.044663536s" totalImagesPullingTime="19.95938102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:33.044200794 +0000 UTC m=+24.208751196" watchObservedRunningTime="2026-07-08 19:42:33.044663536 +0000 UTC m=+24.209213943" Jul 08 19:42:33.541212 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.541082 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T19:42:32.794714232Z","UUID":"fcd14d62-b40a-438a-a5e7-649d66d01be8","Handler":null,"Name":"","Endpoint":""} Jul 08 19:42:33.542785 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.542758 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 19:42:33.542941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.542803 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 19:42:33.668279 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.668245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:33.668461 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.668245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:33.668461 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:33.668387 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:33.668574 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:33.668478 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:33.668574 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:33.668245 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:33.668683 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:33.668606 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:34.027353 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:34.027307 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" event={"ID":"69a75894-7456-4c45-aa91-fa70553e36ca","Type":"ContainerStarted","Data":"f9d54510e28860c230fff25c88a8bed571bda484ed46a19417096c803a10a8c5"} Jul 08 19:42:35.033424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:35.033384 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" event={"ID":"3217a1bd-e7aa-4004-a2cb-3f6b3fa6f337","Type":"ContainerStarted","Data":"f474d1fe0a11f899d251d7158454b3d4797eb74d0632b5a3d2efa47a6fa34f21"} Jul 08 19:42:35.052608 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:35.052546 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pgxdk" podStartSLOduration=3.596025633 podStartE2EDuration="26.052525985s" totalImagesPullingTime="22.456500352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:35.052377591 +0000 UTC m=+26.216927985" watchObservedRunningTime="2026-07-08 19:42:35.052525985 +0000 UTC m=+26.217076391" Jul 08 19:42:35.426598 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:35.426512 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:35.427356 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:35.427340 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:35.667542 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:35.667508 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:35.667712 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:35.667510 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:35.667712 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:35.667624 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:35.667712 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:35.667523 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:35.667712 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:35.667696 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:35.667839 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:35.667783 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:36.041336 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:36.041116 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" event={"ID":"69a75894-7456-4c45-aa91-fa70553e36ca","Type":"ContainerStarted","Data":"02b97f172736781dbf4dfbee500119f97c0c4ece36a183114cff02e4289b48df"} Jul 08 19:42:36.041726 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:36.041551 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:36.041879 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:36.041849 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:36.042007 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:36.041917 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:36.042007 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:36.041964 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:36.042120 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:36.042031 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v4rbh" Jul 08 19:42:36.059130 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:36.059101 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:36.059330 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:36.059256 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:42:36.067690 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:36.067641 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" podStartSLOduration=7.02934164 podStartE2EDuration="27.067626912s" totalImagesPullingTime="20.038285272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:36.067492511 +0000 UTC m=+27.232042914" watchObservedRunningTime="2026-07-08 19:42:36.067626912 +0000 UTC m=+27.232177337" Jul 08 19:42:37.667781 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:37.667749 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:37.668596 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:37.667749 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:37.668596 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:37.667900 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:37.668596 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:37.667971 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:37.668596 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:37.667749 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:37.668596 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:37.668054 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:37.789099 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:37.789060 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pr9w2"] Jul 08 19:42:37.795393 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:37.795223 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q5dlc"] Jul 08 19:42:37.795590 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:37.795566 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z946k"] Jul 08 19:42:38.045969 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:38.045925 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:38.045969 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:38.045943 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:38.045969 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:38.045941 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:38.046302 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:38.046147 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:38.046302 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:38.046228 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:38.046405 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:38.046307 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:39.050328 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:39.050116 2572 generic.go:356] "Generic (PLEG): container finished" podID="392e7cb3-646e-4452-bf3c-0e44dae9bbff" containerID="8750550d5dc010b84b1f264cd1c470cd95f0e363283e7450cfc21b5b51f49c25" exitCode=0 Jul 08 19:42:39.050727 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:39.050213 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-852tk" event={"ID":"392e7cb3-646e-4452-bf3c-0e44dae9bbff","Type":"ContainerDied","Data":"8750550d5dc010b84b1f264cd1c470cd95f0e363283e7450cfc21b5b51f49c25"} Jul 08 19:42:39.668789 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:39.668749 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:39.668970 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:39.668847 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:39.668970 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:39.668879 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:39.668970 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:39.668846 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:39.668970 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:39.668947 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:39.669225 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:39.669013 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:41.056683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:41.056644 2572 generic.go:356] "Generic (PLEG): container finished" podID="392e7cb3-646e-4452-bf3c-0e44dae9bbff" containerID="84a60321691a21912f4a180e6de5a6a0ea5c59b8f5612c0c5e913a5a1e22d9a4" exitCode=0 Jul 08 19:42:41.057094 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:41.056705 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-852tk" event={"ID":"392e7cb3-646e-4452-bf3c-0e44dae9bbff","Type":"ContainerDied","Data":"84a60321691a21912f4a180e6de5a6a0ea5c59b8f5612c0c5e913a5a1e22d9a4"} Jul 08 19:42:41.667840 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:41.667809 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:41.668016 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:41.667809 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:41.668016 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:41.667924 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z946k" podUID="0ca20598-adcb-4c45-87a1-91df3d8a8806" Jul 08 19:42:41.668016 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:41.667809 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:41.668137 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:41.668019 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:42:41.668137 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:41.668067 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5dlc" podUID="d39c7b30-0ef8-46ad-b195-277e67817f64" Jul 08 19:42:42.190683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:42.190647 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:42.190683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:42.190692 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:42.191152 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:42.190719 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwv9\" (UniqueName: \"kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9\") pod \"network-check-target-z946k\" (UID: \"0ca20598-adcb-4c45-87a1-91df3d8a8806\") " pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:42.191152 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:42.190773 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:42.191152 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:42.190810 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:42.191152 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:42.190819 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 19:42:42.191152 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:42.190839 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 19:42:42.191152 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:42.190850 2572 projected.go:196] Error preparing data for projected volume kube-api-access-pbwv9 for pod openshift-network-diagnostics/network-check-target-z946k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:42.191152 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:42.190855 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs podName:cc99e320-6751-4034-9446-70884569cdbb nodeName:}" failed. No retries permitted until 2026-07-08 19:43:14.190832544 +0000 UTC m=+65.355382926 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs") pod "network-metrics-daemon-pr9w2" (UID: "cc99e320-6751-4034-9446-70884569cdbb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 19:42:42.191152 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:42.190875 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret podName:d39c7b30-0ef8-46ad-b195-277e67817f64 nodeName:}" failed. No retries permitted until 2026-07-08 19:43:14.190865547 +0000 UTC m=+65.355415932 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret") pod "global-pull-secret-syncer-q5dlc" (UID: "d39c7b30-0ef8-46ad-b195-277e67817f64") : object "kube-system"/"original-pull-secret" not registered Jul 08 19:42:42.191152 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:42.190892 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9 podName:0ca20598-adcb-4c45-87a1-91df3d8a8806 nodeName:}" failed. No retries permitted until 2026-07-08 19:43:14.190882148 +0000 UTC m=+65.355432529 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pbwv9" (UniqueName: "kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9") pod "network-check-target-z946k" (UID: "0ca20598-adcb-4c45-87a1-91df3d8a8806") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 19:42:43.063834 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.063797 2572 generic.go:356] "Generic (PLEG): container finished" podID="392e7cb3-646e-4452-bf3c-0e44dae9bbff" containerID="862152948d64bc3cd5231a883feab10d1bf60a04292a423c3df058f6ce0dc9d9" exitCode=0 Jul 08 19:42:43.063980 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.063854 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-852tk" event={"ID":"392e7cb3-646e-4452-bf3c-0e44dae9bbff","Type":"ContainerDied","Data":"862152948d64bc3cd5231a883feab10d1bf60a04292a423c3df058f6ce0dc9d9"} Jul 08 19:42:43.604097 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.604069 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-22.ec2.internal" event="NodeReady" Jul 08 19:42:43.604496 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.604206 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 19:42:43.644312 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.644277 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gg7bd"] Jul 08 19:42:43.661791 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.661707 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fnphv"] Jul 08 19:42:43.661950 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.661811 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.664561 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.664374 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 19:42:43.664561 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.664442 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w5s5j\"" Jul 08 19:42:43.664561 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.664449 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 19:42:43.676998 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.676972 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fnphv" Jul 08 19:42:43.680236 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.679906 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 19:42:43.680236 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.679906 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 19:42:43.680236 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.679976 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 19:42:43.680236 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.680218 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4qkct\"" Jul 08 19:42:43.681105 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.681085 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gg7bd"] Jul 08 19:42:43.681224 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.681203 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fnphv"] Jul 08 19:42:43.681349 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.681321 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:42:43.681417 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.681390 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:42:43.681487 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.681473 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:42:43.683560 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.683524 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 19:42:43.683779 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.683716 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 19:42:43.683779 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.683763 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-84p6s\"" Jul 08 19:42:43.684021 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.684000 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 19:42:43.684134 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.684060 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 19:42:43.684201 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.684182 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dpn4j\"" Jul 08 19:42:43.801816 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.801778 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpb6m\" (UniqueName: \"kubernetes.io/projected/6d3f85dc-1bff-4fd1-940e-216b04a813ce-kube-api-access-dpb6m\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.801981 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.801843 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f85dc-1bff-4fd1-940e-216b04a813ce-config-volume\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.801981 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.801899 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49fc0d84-1956-411d-b080-a249b75458f0-cert\") pod \"ingress-canary-fnphv\" (UID: \"49fc0d84-1956-411d-b080-a249b75458f0\") " pod="openshift-ingress-canary/ingress-canary-fnphv" Jul 08 19:42:43.801981 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.801948 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.801981 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.801968 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3f85dc-1bff-4fd1-940e-216b04a813ce-tmp-dir\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.802116 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.802002 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttp8j\" (UniqueName: \"kubernetes.io/projected/49fc0d84-1956-411d-b080-a249b75458f0-kube-api-access-ttp8j\") pod \"ingress-canary-fnphv\" (UID: \"49fc0d84-1956-411d-b080-a249b75458f0\") " pod="openshift-ingress-canary/ingress-canary-fnphv" Jul 08 19:42:43.903108 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.903066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3f85dc-1bff-4fd1-940e-216b04a813ce-tmp-dir\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.903108 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.903120 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttp8j\" (UniqueName: \"kubernetes.io/projected/49fc0d84-1956-411d-b080-a249b75458f0-kube-api-access-ttp8j\") pod \"ingress-canary-fnphv\" (UID: \"49fc0d84-1956-411d-b080-a249b75458f0\") " pod="openshift-ingress-canary/ingress-canary-fnphv" Jul 08 19:42:43.903336 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.903145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpb6m\" (UniqueName: \"kubernetes.io/projected/6d3f85dc-1bff-4fd1-940e-216b04a813ce-kube-api-access-dpb6m\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.903336 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.903212 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f85dc-1bff-4fd1-940e-216b04a813ce-config-volume\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.903336 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.903234 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49fc0d84-1956-411d-b080-a249b75458f0-cert\") pod \"ingress-canary-fnphv\" (UID: \"49fc0d84-1956-411d-b080-a249b75458f0\") " pod="openshift-ingress-canary/ingress-canary-fnphv" Jul 08 19:42:43.903336 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.903258 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.903480 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:43.903347 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:42:43.903480 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:43.903410 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls podName:6d3f85dc-1bff-4fd1-940e-216b04a813ce nodeName:}" failed. No retries permitted until 2026-07-08 19:42:44.40339195 +0000 UTC m=+35.567942333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls") pod "dns-default-gg7bd" (UID: "6d3f85dc-1bff-4fd1-940e-216b04a813ce") : secret "dns-default-metrics-tls" not found Jul 08 19:42:43.903560 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.903504 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d3f85dc-1bff-4fd1-940e-216b04a813ce-tmp-dir\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.903831 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.903813 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6d3f85dc-1bff-4fd1-940e-216b04a813ce-config-volume\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.907670 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.907591 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49fc0d84-1956-411d-b080-a249b75458f0-cert\") pod \"ingress-canary-fnphv\" (UID: \"49fc0d84-1956-411d-b080-a249b75458f0\") " pod="openshift-ingress-canary/ingress-canary-fnphv" Jul 08 19:42:43.911361 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.911337 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpb6m\" (UniqueName: \"kubernetes.io/projected/6d3f85dc-1bff-4fd1-940e-216b04a813ce-kube-api-access-dpb6m\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:43.911518 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.911494 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttp8j\" (UniqueName: \"kubernetes.io/projected/49fc0d84-1956-411d-b080-a249b75458f0-kube-api-access-ttp8j\") pod \"ingress-canary-fnphv\" (UID: \"49fc0d84-1956-411d-b080-a249b75458f0\") " pod="openshift-ingress-canary/ingress-canary-fnphv" Jul 08 19:42:43.989010 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:43.988921 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fnphv" Jul 08 19:42:44.172663 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:44.172628 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fnphv"] Jul 08 19:42:44.177122 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:42:44.177088 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fc0d84_1956_411d_b080_a249b75458f0.slice/crio-4e3c48610e621b114db2026f5226b049edd23133b10f901d40f789deeee8f23c WatchSource:0}: Error finding container 4e3c48610e621b114db2026f5226b049edd23133b10f901d40f789deeee8f23c: Status 404 returned error can't find the container with id 4e3c48610e621b114db2026f5226b049edd23133b10f901d40f789deeee8f23c Jul 08 19:42:44.407680 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:44.407644 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:44.407896 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:44.407811 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:42:44.407896 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:44.407891 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls podName:6d3f85dc-1bff-4fd1-940e-216b04a813ce nodeName:}" failed. No retries permitted until 2026-07-08 19:42:45.407870087 +0000 UTC m=+36.572420482 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls") pod "dns-default-gg7bd" (UID: "6d3f85dc-1bff-4fd1-940e-216b04a813ce") : secret "dns-default-metrics-tls" not found Jul 08 19:42:45.069979 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:45.069907 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fnphv" event={"ID":"49fc0d84-1956-411d-b080-a249b75458f0","Type":"ContainerStarted","Data":"4e3c48610e621b114db2026f5226b049edd23133b10f901d40f789deeee8f23c"} Jul 08 19:42:45.414934 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:45.414851 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:45.415114 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:45.415095 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:42:45.415348 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:45.415325 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls podName:6d3f85dc-1bff-4fd1-940e-216b04a813ce nodeName:}" failed. No retries permitted until 2026-07-08 19:42:47.415289295 +0000 UTC m=+38.579839683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls") pod "dns-default-gg7bd" (UID: "6d3f85dc-1bff-4fd1-940e-216b04a813ce") : secret "dns-default-metrics-tls" not found Jul 08 19:42:47.432301 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:47.432041 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:47.432703 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:47.432236 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:42:47.432703 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:47.432430 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls podName:6d3f85dc-1bff-4fd1-940e-216b04a813ce nodeName:}" failed. No retries permitted until 2026-07-08 19:42:51.43241405 +0000 UTC m=+42.596964437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls") pod "dns-default-gg7bd" (UID: "6d3f85dc-1bff-4fd1-940e-216b04a813ce") : secret "dns-default-metrics-tls" not found Jul 08 19:42:49.081095 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:49.081064 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fnphv" event={"ID":"49fc0d84-1956-411d-b080-a249b75458f0","Type":"ContainerStarted","Data":"f6da895e72b44c15f477a9396444b61a476a990ce2d983010a94aaa282ebf781"} Jul 08 19:42:49.101331 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:49.101134 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fnphv" podStartSLOduration=3.260458088 podStartE2EDuration="6.101119327s" totalImagesPullingTime="2.840661239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:49.100559543 +0000 UTC m=+40.265109964" watchObservedRunningTime="2026-07-08 19:42:49.101119327 +0000 UTC m=+40.265669730" Jul 08 19:42:50.089145 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:50.089107 2572 generic.go:356] "Generic (PLEG): container finished" podID="392e7cb3-646e-4452-bf3c-0e44dae9bbff" containerID="299625fac01d90250f854d7fd642387070d65293152694c1bb8bb2054dc94072" exitCode=0 Jul 08 19:42:50.089627 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:50.089194 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-852tk" event={"ID":"392e7cb3-646e-4452-bf3c-0e44dae9bbff","Type":"ContainerDied","Data":"299625fac01d90250f854d7fd642387070d65293152694c1bb8bb2054dc94072"} Jul 08 19:42:51.093742 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:51.093709 2572 generic.go:356] "Generic (PLEG): container finished" podID="392e7cb3-646e-4452-bf3c-0e44dae9bbff" containerID="b53c4ca2873b6808f5372ed4ef36a553a8d0022aed4e50787e06c81da503aecc" exitCode=0 Jul 08 19:42:51.094230 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:51.093763 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-852tk" event={"ID":"392e7cb3-646e-4452-bf3c-0e44dae9bbff","Type":"ContainerDied","Data":"b53c4ca2873b6808f5372ed4ef36a553a8d0022aed4e50787e06c81da503aecc"} Jul 08 19:42:51.460686 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:51.460593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:51.460832 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:51.460744 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:42:51.460832 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:51.460813 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls podName:6d3f85dc-1bff-4fd1-940e-216b04a813ce nodeName:}" failed. No retries permitted until 2026-07-08 19:42:59.460794251 +0000 UTC m=+50.625344634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls") pod "dns-default-gg7bd" (UID: "6d3f85dc-1bff-4fd1-940e-216b04a813ce") : secret "dns-default-metrics-tls" not found Jul 08 19:42:52.098864 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:52.098824 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-852tk" event={"ID":"392e7cb3-646e-4452-bf3c-0e44dae9bbff","Type":"ContainerStarted","Data":"2edd39b12e5dfbc04f120027823fd3dc7a7da61e62bee53cb479d5eedf8ec430"} Jul 08 19:42:52.125335 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:52.125278 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-852tk" podStartSLOduration=8.293174548 podStartE2EDuration="43.125258613s" totalImagesPullingTime="34.832084065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:42:52.124712339 +0000 UTC m=+43.289262741" watchObservedRunningTime="2026-07-08 19:42:52.125258613 +0000 UTC m=+43.289809017" Jul 08 19:42:59.515702 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:42:59.515651 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:42:59.516092 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:59.515806 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:42:59.516092 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:42:59.515872 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls podName:6d3f85dc-1bff-4fd1-940e-216b04a813ce nodeName:}" failed. No retries permitted until 2026-07-08 19:43:15.515856184 +0000 UTC m=+66.680406564 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls") pod "dns-default-gg7bd" (UID: "6d3f85dc-1bff-4fd1-940e-216b04a813ce") : secret "dns-default-metrics-tls" not found Jul 08 19:43:08.056371 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:08.056343 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tz7p9" Jul 08 19:43:14.217672 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.217607 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:43:14.218208 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.217794 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbwv9\" (UniqueName: \"kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9\") pod \"network-check-target-z946k\" (UID: \"0ca20598-adcb-4c45-87a1-91df3d8a8806\") " pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:43:14.218208 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.217844 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:43:14.220106 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.220083 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 19:43:14.220224 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.220086 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 19:43:14.220406 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.220391 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 19:43:14.228460 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:43:14.228435 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 19:43:14.228581 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:43:14.228514 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs podName:cc99e320-6751-4034-9446-70884569cdbb nodeName:}" failed. No retries permitted until 2026-07-08 19:44:18.228491982 +0000 UTC m=+129.393042366 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs") pod "network-metrics-daemon-pr9w2" (UID: "cc99e320-6751-4034-9446-70884569cdbb") : secret "metrics-daemon-secret" not found Jul 08 19:43:14.230178 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.230148 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 19:43:14.231767 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.231743 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d39c7b30-0ef8-46ad-b195-277e67817f64-original-pull-secret\") pod \"global-pull-secret-syncer-q5dlc\" (UID: \"d39c7b30-0ef8-46ad-b195-277e67817f64\") " pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:43:14.241004 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.240978 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbwv9\" (UniqueName: \"kubernetes.io/projected/0ca20598-adcb-4c45-87a1-91df3d8a8806-kube-api-access-pbwv9\") pod \"network-check-target-z946k\" (UID: \"0ca20598-adcb-4c45-87a1-91df3d8a8806\") " pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:43:14.306406 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.306372 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-84p6s\"" Jul 08 19:43:14.310206 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.310189 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5dlc" Jul 08 19:43:14.315048 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.315021 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:43:14.453047 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.453015 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z946k"] Jul 08 19:43:14.455849 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:43:14.455823 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca20598_adcb_4c45_87a1_91df3d8a8806.slice/crio-f7fd161f81b904d8e8177de8ab146a910a4360df45332645883e82868a4eec95 WatchSource:0}: Error finding container f7fd161f81b904d8e8177de8ab146a910a4360df45332645883e82868a4eec95: Status 404 returned error can't find the container with id f7fd161f81b904d8e8177de8ab146a910a4360df45332645883e82868a4eec95 Jul 08 19:43:14.467832 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:14.467759 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q5dlc"] Jul 08 19:43:14.471753 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:43:14.471716 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39c7b30_0ef8_46ad_b195_277e67817f64.slice/crio-7e5d98a34248a44d1a7e4d583a1b2d12e5aa51f447d452e5d88576a332441637 WatchSource:0}: Error finding container 7e5d98a34248a44d1a7e4d583a1b2d12e5aa51f447d452e5d88576a332441637: Status 404 returned error can't find the container with id 7e5d98a34248a44d1a7e4d583a1b2d12e5aa51f447d452e5d88576a332441637 Jul 08 19:43:15.152459 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:15.152408 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q5dlc" event={"ID":"d39c7b30-0ef8-46ad-b195-277e67817f64","Type":"ContainerStarted","Data":"7e5d98a34248a44d1a7e4d583a1b2d12e5aa51f447d452e5d88576a332441637"} Jul 08 19:43:15.153967 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:15.153905 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z946k" event={"ID":"0ca20598-adcb-4c45-87a1-91df3d8a8806","Type":"ContainerStarted","Data":"f7fd161f81b904d8e8177de8ab146a910a4360df45332645883e82868a4eec95"} Jul 08 19:43:15.528483 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:15.528441 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:43:15.528994 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:43:15.528621 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:43:15.528994 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:43:15.528696 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls podName:6d3f85dc-1bff-4fd1-940e-216b04a813ce nodeName:}" failed. No retries permitted until 2026-07-08 19:43:47.52867484 +0000 UTC m=+98.693225235 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls") pod "dns-default-gg7bd" (UID: "6d3f85dc-1bff-4fd1-940e-216b04a813ce") : secret "dns-default-metrics-tls" not found Jul 08 19:43:20.168515 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:20.168476 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z946k" event={"ID":"0ca20598-adcb-4c45-87a1-91df3d8a8806","Type":"ContainerStarted","Data":"162339fcd674205c23834e15e4cf8a999569530fe8833840740f5c85b019dce4"} Jul 08 19:43:20.168975 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:20.168562 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:43:20.169812 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:20.169790 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q5dlc" event={"ID":"d39c7b30-0ef8-46ad-b195-277e67817f64","Type":"ContainerStarted","Data":"35abc083f8a8459550744bbd7ae04bcc8eb9efaa1ccf1a055f2218f7dd72f12d"} Jul 08 19:43:20.185213 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:20.185127 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z946k" podStartSLOduration=66.137895081 podStartE2EDuration="1m11.185110906s" totalImagesPullingTime="5.047215825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:43:20.185001405 +0000 UTC m=+71.349551809" watchObservedRunningTime="2026-07-08 19:43:20.185110906 +0000 UTC m=+71.349661308" Jul 08 19:43:20.199303 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:20.199234 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q5dlc" podStartSLOduration=66.167529203 podStartE2EDuration="1m11.199212681s" totalImagesPullingTime="5.031683478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:43:20.198579299 +0000 UTC m=+71.363129721" watchObservedRunningTime="2026-07-08 19:43:20.199212681 +0000 UTC m=+71.363763085" Jul 08 19:43:47.542329 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:47.542291 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:43:47.542731 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:43:47.542404 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:43:47.542731 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:43:47.542462 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls podName:6d3f85dc-1bff-4fd1-940e-216b04a813ce nodeName:}" failed. No retries permitted until 2026-07-08 19:44:51.542449308 +0000 UTC m=+162.706999689 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls") pod "dns-default-gg7bd" (UID: "6d3f85dc-1bff-4fd1-940e-216b04a813ce") : secret "dns-default-metrics-tls" not found Jul 08 19:43:51.174776 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:43:51.174746 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z946k" Jul 08 19:44:18.245643 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:44:18.245593 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:44:18.246201 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:44:18.245758 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 19:44:18.246201 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:44:18.245831 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs podName:cc99e320-6751-4034-9446-70884569cdbb nodeName:}" failed. No retries permitted until 2026-07-08 19:46:20.245816303 +0000 UTC m=+251.410366715 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs") pod "network-metrics-daemon-pr9w2" (UID: "cc99e320-6751-4034-9446-70884569cdbb") : secret "metrics-daemon-secret" not found Jul 08 19:44:46.673501 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:44:46.673447 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gg7bd" podUID="6d3f85dc-1bff-4fd1-940e-216b04a813ce" Jul 08 19:44:46.696742 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:44:46.696716 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:44:47.376825 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:44:47.376798 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gg7bd" Jul 08 19:44:51.561931 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:44:51.561889 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:44:51.562448 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:44:51.562026 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 19:44:51.562448 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:44:51.562104 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls podName:6d3f85dc-1bff-4fd1-940e-216b04a813ce nodeName:}" failed. No retries permitted until 2026-07-08 19:46:53.562084367 +0000 UTC m=+284.726634769 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls") pod "dns-default-gg7bd" (UID: "6d3f85dc-1bff-4fd1-940e-216b04a813ce") : secret "dns-default-metrics-tls" not found Jul 08 19:44:59.668986 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:44:59.668942 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:46:05.260213 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.260152 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk"] Jul 08 19:46:05.262458 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.262435 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw"] Jul 08 19:46:05.262584 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.262567 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:05.264452 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.264435 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.267261 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.267241 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 08 19:46:05.267454 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.267440 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 08 19:46:05.268007 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.267988 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:46:05.268230 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.268210 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 19:46:05.268230 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.268224 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m2rkq\"" Jul 08 19:46:05.268360 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.268217 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:46:05.268360 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.268224 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w78xq\"" Jul 08 19:46:05.268360 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.268211 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 19:46:05.268360 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.268227 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 08 19:46:05.282672 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.282649 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk"] Jul 08 19:46:05.285839 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.285809 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw"] Jul 08 19:46:05.350784 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.350749 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ea05414-ca40-4d7e-a66d-40dd97d8a2a7-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-b8wqw\" (UID: \"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.350784 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.350786 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ea05414-ca40-4d7e-a66d-40dd97d8a2a7-config\") pod \"service-ca-operator-5d46d9ffd4-b8wqw\" (UID: \"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.351043 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.350818 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:05.351043 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.350899 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7bg\" (UniqueName: \"kubernetes.io/projected/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-kube-api-access-cs7bg\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:05.351043 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.350937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5mh\" (UniqueName: \"kubernetes.io/projected/4ea05414-ca40-4d7e-a66d-40dd97d8a2a7-kube-api-access-nj5mh\") pod \"service-ca-operator-5d46d9ffd4-b8wqw\" (UID: \"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.355030 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.355006 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z"] Jul 08 19:46:05.357052 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.357037 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.358544 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.358521 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-l7rg9"] Jul 08 19:46:05.359631 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.359613 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:46:05.360264 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.360247 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lzh48\"" Jul 08 19:46:05.360337 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.360264 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ll9pj"] Jul 08 19:46:05.360396 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.360381 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.360591 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.360521 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 19:46:05.362033 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.362014 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.363442 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.363417 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 19:46:05.363527 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.363424 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 19:46:05.363589 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.363571 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 19:46:05.363700 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.363684 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 19:46:05.363758 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.363685 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:46:05.363797 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.363763 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 19:46:05.363839 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.363791 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dlqtm\"" Jul 08 19:46:05.364221 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.364203 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 19:46:05.364299 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.364244 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 08 19:46:05.364544 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.364521 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 08 19:46:05.364728 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.364715 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 19:46:05.364728 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.364731 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jwhkg\"" Jul 08 19:46:05.370248 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.370013 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 19:46:05.372279 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.372255 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z"] Jul 08 19:46:05.375051 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.374987 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-l7rg9"] Jul 08 19:46:05.377522 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.376763 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ll9pj"] Jul 08 19:46:05.377522 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.377170 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 08 19:46:05.451555 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451518 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-trusted-ca\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.451555 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451558 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:05.451804 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451579 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b68237ef-7e03-4ded-a8e2-3e69518b6736-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.451804 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451608 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7bg\" (UniqueName: \"kubernetes.io/projected/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-kube-api-access-cs7bg\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:05.451804 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451634 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5mh\" (UniqueName: \"kubernetes.io/projected/4ea05414-ca40-4d7e-a66d-40dd97d8a2a7-kube-api-access-nj5mh\") pod \"service-ca-operator-5d46d9ffd4-b8wqw\" (UID: \"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.451804 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451670 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-config\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.451804 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:05.451694 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:46:05.451804 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451723 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b68237ef-7e03-4ded-a8e2-3e69518b6736-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.451804 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451786 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b68237ef-7e03-4ded-a8e2-3e69518b6736-tmp\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.451804 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:05.451802 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls podName:e06a34ad-deaa-4b6e-9d0f-2da8d006f88f nodeName:}" failed. No retries permitted until 2026-07-08 19:46:05.951778546 +0000 UTC m=+237.116328927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nnmvk" (UID: "e06a34ad-deaa-4b6e-9d0f-2da8d006f88f") : secret "samples-operator-tls" not found Jul 08 19:46:05.452280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451868 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ea05414-ca40-4d7e-a66d-40dd97d8a2a7-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-b8wqw\" (UID: \"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.452280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451886 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ea05414-ca40-4d7e-a66d-40dd97d8a2a7-config\") pod \"service-ca-operator-5d46d9ffd4-b8wqw\" (UID: \"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.452280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451906 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-serving-cert\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.452280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451921 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmwq\" (UniqueName: \"kubernetes.io/projected/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-kube-api-access-tqmwq\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.452280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.451993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4617b8-a240-411a-a380-0408e88ed3f1-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-x4b9z\" (UID: \"2e4617b8-a240-411a-a380-0408e88ed3f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.452280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.452020 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2e4617b8-a240-411a-a380-0408e88ed3f1-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-x4b9z\" (UID: \"2e4617b8-a240-411a-a380-0408e88ed3f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.452280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.452054 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86qsm\" (UniqueName: \"kubernetes.io/projected/2e4617b8-a240-411a-a380-0408e88ed3f1-kube-api-access-86qsm\") pod \"kube-storage-version-migrator-operator-846bf668f7-x4b9z\" (UID: \"2e4617b8-a240-411a-a380-0408e88ed3f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.452280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.452097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b68237ef-7e03-4ded-a8e2-3e69518b6736-snapshots\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.452280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.452120 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b68237ef-7e03-4ded-a8e2-3e69518b6736-serving-cert\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.452280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.452138 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vckfj\" (UniqueName: \"kubernetes.io/projected/b68237ef-7e03-4ded-a8e2-3e69518b6736-kube-api-access-vckfj\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.452654 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.452556 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ea05414-ca40-4d7e-a66d-40dd97d8a2a7-config\") pod \"service-ca-operator-5d46d9ffd4-b8wqw\" (UID: \"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.454197 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.454178 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ea05414-ca40-4d7e-a66d-40dd97d8a2a7-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-b8wqw\" (UID: \"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.457021 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.456994 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-74d89d68cf-sjvht"] Jul 08 19:46:05.459516 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.459497 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.461730 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.461712 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 19:46:05.461824 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.461800 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 19:46:05.462619 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.462600 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 19:46:05.462725 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.462624 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 19:46:05.462725 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.462657 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 19:46:05.462991 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.462975 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4724n\"" Jul 08 19:46:05.463572 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.463558 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 19:46:05.472456 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.472431 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5mh\" (UniqueName: \"kubernetes.io/projected/4ea05414-ca40-4d7e-a66d-40dd97d8a2a7-kube-api-access-nj5mh\") pod \"service-ca-operator-5d46d9ffd4-b8wqw\" (UID: \"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.474387 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.474364 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7bg\" (UniqueName: \"kubernetes.io/projected/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-kube-api-access-cs7bg\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:05.481043 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.481022 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74d89d68cf-sjvht"] Jul 08 19:46:05.552587 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.552558 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4617b8-a240-411a-a380-0408e88ed3f1-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-x4b9z\" (UID: \"2e4617b8-a240-411a-a380-0408e88ed3f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.552769 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.552592 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2e4617b8-a240-411a-a380-0408e88ed3f1-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-x4b9z\" (UID: \"2e4617b8-a240-411a-a380-0408e88ed3f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.552769 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.552619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-trusted-ca\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.552769 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.552731 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-default-certificate\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.552928 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.552789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b68237ef-7e03-4ded-a8e2-3e69518b6736-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.552928 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.552829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-serving-cert\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.552928 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.552873 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.553087 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.552925 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rjw5\" (UniqueName: \"kubernetes.io/projected/43c4d634-0090-4a22-acd1-51deeaa3584e-kube-api-access-4rjw5\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.553087 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553000 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qsm\" (UniqueName: \"kubernetes.io/projected/2e4617b8-a240-411a-a380-0408e88ed3f1-kube-api-access-86qsm\") pod \"kube-storage-version-migrator-operator-846bf668f7-x4b9z\" (UID: \"2e4617b8-a240-411a-a380-0408e88ed3f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.553087 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b68237ef-7e03-4ded-a8e2-3e69518b6736-snapshots\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.553087 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553077 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b68237ef-7e03-4ded-a8e2-3e69518b6736-serving-cert\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.553318 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553121 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.553318 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vckfj\" (UniqueName: \"kubernetes.io/projected/b68237ef-7e03-4ded-a8e2-3e69518b6736-kube-api-access-vckfj\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.553318 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553229 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b68237ef-7e03-4ded-a8e2-3e69518b6736-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.553318 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553273 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-config\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.553318 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553286 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2e4617b8-a240-411a-a380-0408e88ed3f1-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-x4b9z\" (UID: \"2e4617b8-a240-411a-a380-0408e88ed3f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.553318 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553306 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b68237ef-7e03-4ded-a8e2-3e69518b6736-tmp\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.553608 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553348 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-stats-auth\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.553709 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553688 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-trusted-ca\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.553775 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553748 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b68237ef-7e03-4ded-a8e2-3e69518b6736-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.553824 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.553806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmwq\" (UniqueName: \"kubernetes.io/projected/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-kube-api-access-tqmwq\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.554144 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.554123 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b68237ef-7e03-4ded-a8e2-3e69518b6736-tmp\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.554322 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.554230 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b68237ef-7e03-4ded-a8e2-3e69518b6736-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.554322 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.554264 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b68237ef-7e03-4ded-a8e2-3e69518b6736-snapshots\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.554534 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.554513 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-config\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.555503 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.555486 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e4617b8-a240-411a-a380-0408e88ed3f1-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-x4b9z\" (UID: \"2e4617b8-a240-411a-a380-0408e88ed3f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.555660 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.555644 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-serving-cert\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.556039 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.556017 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b68237ef-7e03-4ded-a8e2-3e69518b6736-serving-cert\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.563443 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.563415 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmwq\" (UniqueName: \"kubernetes.io/projected/d2bf9f7a-9465-4c1c-8a14-4a9d39729d03-kube-api-access-tqmwq\") pod \"console-operator-68b757fb67-l7rg9\" (UID: \"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03\") " pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.563531 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.563465 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86qsm\" (UniqueName: \"kubernetes.io/projected/2e4617b8-a240-411a-a380-0408e88ed3f1-kube-api-access-86qsm\") pod \"kube-storage-version-migrator-operator-846bf668f7-x4b9z\" (UID: \"2e4617b8-a240-411a-a380-0408e88ed3f1\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.563682 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.563666 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vckfj\" (UniqueName: \"kubernetes.io/projected/b68237ef-7e03-4ded-a8e2-3e69518b6736-kube-api-access-vckfj\") pod \"insights-operator-5c847dcfc4-ll9pj\" (UID: \"b68237ef-7e03-4ded-a8e2-3e69518b6736\") " pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.577586 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.577561 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" Jul 08 19:46:05.655240 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.654798 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-default-certificate\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.655240 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.654861 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.655240 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.654886 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjw5\" (UniqueName: \"kubernetes.io/projected/43c4d634-0090-4a22-acd1-51deeaa3584e-kube-api-access-4rjw5\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.655240 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.654924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.655240 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.654976 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-stats-auth\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.655517 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:05.655464 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle podName:43c4d634-0090-4a22-acd1-51deeaa3584e nodeName:}" failed. No retries permitted until 2026-07-08 19:46:06.155439398 +0000 UTC m=+237.319989796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle") pod "router-default-74d89d68cf-sjvht" (UID: "43c4d634-0090-4a22-acd1-51deeaa3584e") : configmap references non-existent config key: service-ca.crt Jul 08 19:46:05.655517 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:05.655510 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:46:05.655619 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:05.655551 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs podName:43c4d634-0090-4a22-acd1-51deeaa3584e nodeName:}" failed. No retries permitted until 2026-07-08 19:46:06.155540004 +0000 UTC m=+237.320090399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs") pod "router-default-74d89d68cf-sjvht" (UID: "43c4d634-0090-4a22-acd1-51deeaa3584e") : secret "router-metrics-certs-default" not found Jul 08 19:46:05.657920 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.657891 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-stats-auth\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.657920 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.657905 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-default-certificate\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.668956 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.668930 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" Jul 08 19:46:05.672617 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.672598 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjw5\" (UniqueName: \"kubernetes.io/projected/43c4d634-0090-4a22-acd1-51deeaa3584e-kube-api-access-4rjw5\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:05.680967 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.680940 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:05.686960 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.686937 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" Jul 08 19:46:05.702737 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.702706 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw"] Jul 08 19:46:05.705536 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:05.705493 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea05414_ca40_4d7e_a66d_40dd97d8a2a7.slice/crio-2163ab22089436f7bf4167e786bd25fcd046a677a4a57698583dc80a7f095935 WatchSource:0}: Error finding container 2163ab22089436f7bf4167e786bd25fcd046a677a4a57698583dc80a7f095935: Status 404 returned error can't find the container with id 2163ab22089436f7bf4167e786bd25fcd046a677a4a57698583dc80a7f095935 Jul 08 19:46:05.813479 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.813406 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z"] Jul 08 19:46:05.816570 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:05.816542 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4617b8_a240_411a_a380_0408e88ed3f1.slice/crio-91b15a1517df71e616c8d956f39371ec4cc9f7bb02a38b79e9e98ce957b09a2c WatchSource:0}: Error finding container 91b15a1517df71e616c8d956f39371ec4cc9f7bb02a38b79e9e98ce957b09a2c: Status 404 returned error can't find the container with id 91b15a1517df71e616c8d956f39371ec4cc9f7bb02a38b79e9e98ce957b09a2c Jul 08 19:46:05.957411 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:05.957379 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:05.957582 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:05.957490 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:46:05.957582 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:05.957543 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls podName:e06a34ad-deaa-4b6e-9d0f-2da8d006f88f nodeName:}" failed. No retries permitted until 2026-07-08 19:46:06.957529689 +0000 UTC m=+238.122080070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nnmvk" (UID: "e06a34ad-deaa-4b6e-9d0f-2da8d006f88f") : secret "samples-operator-tls" not found Jul 08 19:46:06.038759 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:06.038721 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-l7rg9"] Jul 08 19:46:06.041413 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:06.041352 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2bf9f7a_9465_4c1c_8a14_4a9d39729d03.slice/crio-4e764f25ca3d927dccdd97883d7eaea0e592b7d3dd6564e8d2a06a1a0d1eff1c WatchSource:0}: Error finding container 4e764f25ca3d927dccdd97883d7eaea0e592b7d3dd6564e8d2a06a1a0d1eff1c: Status 404 returned error can't find the container with id 4e764f25ca3d927dccdd97883d7eaea0e592b7d3dd6564e8d2a06a1a0d1eff1c Jul 08 19:46:06.041715 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:06.041695 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ll9pj"] Jul 08 19:46:06.044743 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:06.044721 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68237ef_7e03_4ded_a8e2_3e69518b6736.slice/crio-f84a4c604f706ffb943f39ce082f989287c7c7dccabcb869351a1f557ccbac08 WatchSource:0}: Error finding container f84a4c604f706ffb943f39ce082f989287c7c7dccabcb869351a1f557ccbac08: Status 404 returned error can't find the container with id f84a4c604f706ffb943f39ce082f989287c7c7dccabcb869351a1f557ccbac08 Jul 08 19:46:06.158362 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:06.158265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:06.158362 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:06.158311 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:06.158599 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:06.158441 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle podName:43c4d634-0090-4a22-acd1-51deeaa3584e nodeName:}" failed. No retries permitted until 2026-07-08 19:46:07.158425438 +0000 UTC m=+238.322975836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle") pod "router-default-74d89d68cf-sjvht" (UID: "43c4d634-0090-4a22-acd1-51deeaa3584e") : configmap references non-existent config key: service-ca.crt Jul 08 19:46:06.158599 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:06.158443 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:46:06.158599 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:06.158506 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs podName:43c4d634-0090-4a22-acd1-51deeaa3584e nodeName:}" failed. No retries permitted until 2026-07-08 19:46:07.158489485 +0000 UTC m=+238.323039866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs") pod "router-default-74d89d68cf-sjvht" (UID: "43c4d634-0090-4a22-acd1-51deeaa3584e") : secret "router-metrics-certs-default" not found Jul 08 19:46:06.558659 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:06.558621 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" event={"ID":"2e4617b8-a240-411a-a380-0408e88ed3f1","Type":"ContainerStarted","Data":"91b15a1517df71e616c8d956f39371ec4cc9f7bb02a38b79e9e98ce957b09a2c"} Jul 08 19:46:06.559598 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:06.559576 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" event={"ID":"b68237ef-7e03-4ded-a8e2-3e69518b6736","Type":"ContainerStarted","Data":"f84a4c604f706ffb943f39ce082f989287c7c7dccabcb869351a1f557ccbac08"} Jul 08 19:46:06.560416 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:06.560397 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" event={"ID":"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03","Type":"ContainerStarted","Data":"4e764f25ca3d927dccdd97883d7eaea0e592b7d3dd6564e8d2a06a1a0d1eff1c"} Jul 08 19:46:06.561365 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:06.561345 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" event={"ID":"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7","Type":"ContainerStarted","Data":"2163ab22089436f7bf4167e786bd25fcd046a677a4a57698583dc80a7f095935"} Jul 08 19:46:06.964213 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:06.964101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:06.964371 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:06.964291 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:46:06.964440 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:06.964371 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls podName:e06a34ad-deaa-4b6e-9d0f-2da8d006f88f nodeName:}" failed. No retries permitted until 2026-07-08 19:46:08.964349972 +0000 UTC m=+240.128900379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nnmvk" (UID: "e06a34ad-deaa-4b6e-9d0f-2da8d006f88f") : secret "samples-operator-tls" not found Jul 08 19:46:07.166414 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:07.166262 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:07.166414 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:07.166322 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:07.166414 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:07.166408 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:46:07.166694 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:07.166476 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs podName:43c4d634-0090-4a22-acd1-51deeaa3584e nodeName:}" failed. No retries permitted until 2026-07-08 19:46:09.166458568 +0000 UTC m=+240.331008954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs") pod "router-default-74d89d68cf-sjvht" (UID: "43c4d634-0090-4a22-acd1-51deeaa3584e") : secret "router-metrics-certs-default" not found Jul 08 19:46:07.166694 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:07.166508 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle podName:43c4d634-0090-4a22-acd1-51deeaa3584e nodeName:}" failed. No retries permitted until 2026-07-08 19:46:09.16648763 +0000 UTC m=+240.331038032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle") pod "router-default-74d89d68cf-sjvht" (UID: "43c4d634-0090-4a22-acd1-51deeaa3584e") : configmap references non-existent config key: service-ca.crt Jul 08 19:46:08.984132 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:08.984097 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:08.984594 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:08.984287 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:46:08.984594 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:08.984370 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls podName:e06a34ad-deaa-4b6e-9d0f-2da8d006f88f nodeName:}" failed. No retries permitted until 2026-07-08 19:46:12.984348171 +0000 UTC m=+244.148898568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nnmvk" (UID: "e06a34ad-deaa-4b6e-9d0f-2da8d006f88f") : secret "samples-operator-tls" not found Jul 08 19:46:09.185120 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:09.185076 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:09.185326 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:09.185138 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:09.185326 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:09.185261 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:46:09.185326 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:09.185293 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle podName:43c4d634-0090-4a22-acd1-51deeaa3584e nodeName:}" failed. No retries permitted until 2026-07-08 19:46:13.185271787 +0000 UTC m=+244.349822168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle") pod "router-default-74d89d68cf-sjvht" (UID: "43c4d634-0090-4a22-acd1-51deeaa3584e") : configmap references non-existent config key: service-ca.crt Jul 08 19:46:09.185326 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:09.185323 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs podName:43c4d634-0090-4a22-acd1-51deeaa3584e nodeName:}" failed. No retries permitted until 2026-07-08 19:46:13.185306854 +0000 UTC m=+244.349857246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs") pod "router-default-74d89d68cf-sjvht" (UID: "43c4d634-0090-4a22-acd1-51deeaa3584e") : secret "router-metrics-certs-default" not found Jul 08 19:46:10.574893 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.574853 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" event={"ID":"b68237ef-7e03-4ded-a8e2-3e69518b6736","Type":"ContainerStarted","Data":"49f6441a4f3a65085c4c11a55d2e7361b71d00b51243d90aaf47d4b5d80e7a35"} Jul 08 19:46:10.576561 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.576535 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/0.log" Jul 08 19:46:10.576708 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.576580 2572 generic.go:356] "Generic (PLEG): container finished" podID="d2bf9f7a-9465-4c1c-8a14-4a9d39729d03" containerID="5baf929998b25b5d5ec2769397db3af0ffe590648a72c74358ce038b85a2706f" exitCode=255 Jul 08 19:46:10.576708 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.576667 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" event={"ID":"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03","Type":"ContainerDied","Data":"5baf929998b25b5d5ec2769397db3af0ffe590648a72c74358ce038b85a2706f"} Jul 08 19:46:10.576918 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.576900 2572 scope.go:117] "RemoveContainer" containerID="5baf929998b25b5d5ec2769397db3af0ffe590648a72c74358ce038b85a2706f" Jul 08 19:46:10.578217 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.578185 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" event={"ID":"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7","Type":"ContainerStarted","Data":"063ab80aacb503738395b8786c308685aaf70a14ddc0c8eb683b13c61154474c"} Jul 08 19:46:10.579728 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.579703 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" event={"ID":"2e4617b8-a240-411a-a380-0408e88ed3f1","Type":"ContainerStarted","Data":"7c18e65089dc0f4e1b47fc99af45b9d0fce7e44fc8fbb700160fc345637489cc"} Jul 08 19:46:10.590875 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.590828 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" podStartSLOduration=1.8260746239999999 podStartE2EDuration="5.590814652s" totalImagesPullingTime="3.764740028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:10.590300879 +0000 UTC m=+241.754851283" watchObservedRunningTime="2026-07-08 19:46:10.590814652 +0000 UTC m=+241.755365056" Jul 08 19:46:10.604010 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.603960 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" podStartSLOduration=1.5050663549999999 podStartE2EDuration="5.603941987s" totalImagesPullingTime="4.098875632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:10.603224791 +0000 UTC m=+241.767775195" watchObservedRunningTime="2026-07-08 19:46:10.603941987 +0000 UTC m=+241.768492391" Jul 08 19:46:10.621976 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.621751 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" podStartSLOduration=1.622040538 podStartE2EDuration="5.621728496s" totalImagesPullingTime="3.999687958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:10.620581797 +0000 UTC m=+241.785132201" watchObservedRunningTime="2026-07-08 19:46:10.621728496 +0000 UTC m=+241.786278902" Jul 08 19:46:10.708623 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.708588 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-srckk"] Jul 08 19:46:10.711452 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.711183 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.713717 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.713693 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 19:46:10.713717 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.713710 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 19:46:10.714082 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.714037 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q8lf2\"" Jul 08 19:46:10.724703 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.724678 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-srckk"] Jul 08 19:46:10.798715 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.798679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.798883 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.798733 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ea71571-304d-4d37-83fa-50bf418aab60-data-volume\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.798883 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.798773 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.798883 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.798811 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdglb\" (UniqueName: \"kubernetes.io/projected/6ea71571-304d-4d37-83fa-50bf418aab60-kube-api-access-rdglb\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.798883 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.798829 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6ea71571-304d-4d37-83fa-50bf418aab60-crio-socket\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.899890 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.899810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ea71571-304d-4d37-83fa-50bf418aab60-data-volume\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.899890 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.899843 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.899890 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.899864 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdglb\" (UniqueName: \"kubernetes.io/projected/6ea71571-304d-4d37-83fa-50bf418aab60-kube-api-access-rdglb\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.900123 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:10.899935 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:46:10.900123 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:10.899998 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm podName:6ea71571-304d-4d37-83fa-50bf418aab60 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:11.399981652 +0000 UTC m=+242.564532037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srckk" (UID: "6ea71571-304d-4d37-83fa-50bf418aab60") : configmap "kube-rbac-proxy" not found Jul 08 19:46:10.900123 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.900045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6ea71571-304d-4d37-83fa-50bf418aab60-crio-socket\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.900281 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.900127 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6ea71571-304d-4d37-83fa-50bf418aab60-crio-socket\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.900281 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.900191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.900281 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.900232 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ea71571-304d-4d37-83fa-50bf418aab60-data-volume\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:10.900281 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:10.900278 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:46:10.900403 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:10.900324 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls podName:6ea71571-304d-4d37-83fa-50bf418aab60 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:11.40031018 +0000 UTC m=+242.564860579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srckk" (UID: "6ea71571-304d-4d37-83fa-50bf418aab60") : secret "insights-runtime-extractor-tls" not found Jul 08 19:46:10.911898 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:10.911873 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdglb\" (UniqueName: \"kubernetes.io/projected/6ea71571-304d-4d37-83fa-50bf418aab60-kube-api-access-rdglb\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:11.404365 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:11.404319 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:11.404532 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:11.404383 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:11.404532 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:11.404449 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:46:11.404532 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:11.404470 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:46:11.404532 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:11.404514 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm podName:6ea71571-304d-4d37-83fa-50bf418aab60 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:12.404499137 +0000 UTC m=+243.569049518 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srckk" (UID: "6ea71571-304d-4d37-83fa-50bf418aab60") : configmap "kube-rbac-proxy" not found Jul 08 19:46:11.404532 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:11.404528 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls podName:6ea71571-304d-4d37-83fa-50bf418aab60 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:12.404521729 +0000 UTC m=+243.569072110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srckk" (UID: "6ea71571-304d-4d37-83fa-50bf418aab60") : secret "insights-runtime-extractor-tls" not found Jul 08 19:46:11.583585 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:11.583560 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 19:46:11.584007 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:11.583992 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/0.log" Jul 08 19:46:11.584082 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:11.584026 2572 generic.go:356] "Generic (PLEG): container finished" podID="d2bf9f7a-9465-4c1c-8a14-4a9d39729d03" containerID="bb95337761170b63f90eceed664d3ca807bd21e65cd711bf9136d2832143d72d" exitCode=255 Jul 08 19:46:11.584178 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:11.584136 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" event={"ID":"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03","Type":"ContainerDied","Data":"bb95337761170b63f90eceed664d3ca807bd21e65cd711bf9136d2832143d72d"} Jul 08 19:46:11.584241 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:11.584216 2572 scope.go:117] "RemoveContainer" containerID="5baf929998b25b5d5ec2769397db3af0ffe590648a72c74358ce038b85a2706f" Jul 08 19:46:11.584427 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:11.584407 2572 scope.go:117] "RemoveContainer" containerID="bb95337761170b63f90eceed664d3ca807bd21e65cd711bf9136d2832143d72d" Jul 08 19:46:11.584641 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:11.584619 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-l7rg9_openshift-console-operator(d2bf9f7a-9465-4c1c-8a14-4a9d39729d03)\"" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" podUID="d2bf9f7a-9465-4c1c-8a14-4a9d39729d03" Jul 08 19:46:12.414142 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.414108 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:12.414309 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.414183 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:12.414309 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:12.414255 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:46:12.414309 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:12.414260 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:46:12.414431 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:12.414320 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm podName:6ea71571-304d-4d37-83fa-50bf418aab60 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:14.414306452 +0000 UTC m=+245.578856832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srckk" (UID: "6ea71571-304d-4d37-83fa-50bf418aab60") : configmap "kube-rbac-proxy" not found Jul 08 19:46:12.414431 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:12.414335 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls podName:6ea71571-304d-4d37-83fa-50bf418aab60 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:14.414329439 +0000 UTC m=+245.578879820 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srckk" (UID: "6ea71571-304d-4d37-83fa-50bf418aab60") : secret "insights-runtime-extractor-tls" not found Jul 08 19:46:12.588098 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.588070 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 19:46:12.588501 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.588405 2572 scope.go:117] "RemoveContainer" containerID="bb95337761170b63f90eceed664d3ca807bd21e65cd711bf9136d2832143d72d" Jul 08 19:46:12.588578 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:12.588559 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-l7rg9_openshift-console-operator(d2bf9f7a-9465-4c1c-8a14-4a9d39729d03)\"" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" podUID="d2bf9f7a-9465-4c1c-8a14-4a9d39729d03" Jul 08 19:46:12.698400 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.698321 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jx9bx_53cf257b-7924-4126-9711-b432a663371a/dns-node-resolver/0.log" Jul 08 19:46:12.903586 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.903552 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-f6w56"] Jul 08 19:46:12.906318 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.906299 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:12.908594 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.908573 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 08 19:46:12.908594 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.908585 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 08 19:46:12.908834 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.908617 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 08 19:46:12.908891 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.908853 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-s7krc\"" Jul 08 19:46:12.908931 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.908912 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 08 19:46:12.913336 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:12.913306 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-f6w56"] Jul 08 19:46:13.020391 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.020358 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97eb1c0a-6d15-401b-b287-5e3fc9009fa0-signing-cabundle\") pod \"service-ca-758f599c5-f6w56\" (UID: \"97eb1c0a-6d15-401b-b287-5e3fc9009fa0\") " pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:13.020565 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.020419 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97eb1c0a-6d15-401b-b287-5e3fc9009fa0-signing-key\") pod \"service-ca-758f599c5-f6w56\" (UID: \"97eb1c0a-6d15-401b-b287-5e3fc9009fa0\") " pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:13.020565 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.020448 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dpbp\" (UniqueName: \"kubernetes.io/projected/97eb1c0a-6d15-401b-b287-5e3fc9009fa0-kube-api-access-6dpbp\") pod \"service-ca-758f599c5-f6w56\" (UID: \"97eb1c0a-6d15-401b-b287-5e3fc9009fa0\") " pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:13.020565 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.020506 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:13.020676 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:13.020634 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 19:46:13.020710 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:13.020684 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls podName:e06a34ad-deaa-4b6e-9d0f-2da8d006f88f nodeName:}" failed. No retries permitted until 2026-07-08 19:46:21.020669973 +0000 UTC m=+252.185220353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nnmvk" (UID: "e06a34ad-deaa-4b6e-9d0f-2da8d006f88f") : secret "samples-operator-tls" not found Jul 08 19:46:13.121259 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.121221 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97eb1c0a-6d15-401b-b287-5e3fc9009fa0-signing-cabundle\") pod \"service-ca-758f599c5-f6w56\" (UID: \"97eb1c0a-6d15-401b-b287-5e3fc9009fa0\") " pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:13.121429 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.121279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97eb1c0a-6d15-401b-b287-5e3fc9009fa0-signing-key\") pod \"service-ca-758f599c5-f6w56\" (UID: \"97eb1c0a-6d15-401b-b287-5e3fc9009fa0\") " pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:13.121429 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.121297 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dpbp\" (UniqueName: \"kubernetes.io/projected/97eb1c0a-6d15-401b-b287-5e3fc9009fa0-kube-api-access-6dpbp\") pod \"service-ca-758f599c5-f6w56\" (UID: \"97eb1c0a-6d15-401b-b287-5e3fc9009fa0\") " pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:13.121958 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.121931 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97eb1c0a-6d15-401b-b287-5e3fc9009fa0-signing-cabundle\") pod \"service-ca-758f599c5-f6w56\" (UID: \"97eb1c0a-6d15-401b-b287-5e3fc9009fa0\") " pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:13.123591 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.123570 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97eb1c0a-6d15-401b-b287-5e3fc9009fa0-signing-key\") pod \"service-ca-758f599c5-f6w56\" (UID: \"97eb1c0a-6d15-401b-b287-5e3fc9009fa0\") " pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:13.130104 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.130085 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dpbp\" (UniqueName: \"kubernetes.io/projected/97eb1c0a-6d15-401b-b287-5e3fc9009fa0-kube-api-access-6dpbp\") pod \"service-ca-758f599c5-f6w56\" (UID: \"97eb1c0a-6d15-401b-b287-5e3fc9009fa0\") " pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:13.215546 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.215512 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-f6w56" Jul 08 19:46:13.221805 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.221778 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:13.221917 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.221834 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:13.221917 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:13.221875 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 19:46:13.222005 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:13.221932 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs podName:43c4d634-0090-4a22-acd1-51deeaa3584e nodeName:}" failed. No retries permitted until 2026-07-08 19:46:21.221914595 +0000 UTC m=+252.386464995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs") pod "router-default-74d89d68cf-sjvht" (UID: "43c4d634-0090-4a22-acd1-51deeaa3584e") : secret "router-metrics-certs-default" not found Jul 08 19:46:13.222046 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:13.222000 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle podName:43c4d634-0090-4a22-acd1-51deeaa3584e nodeName:}" failed. No retries permitted until 2026-07-08 19:46:21.221985466 +0000 UTC m=+252.386535850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle") pod "router-default-74d89d68cf-sjvht" (UID: "43c4d634-0090-4a22-acd1-51deeaa3584e") : configmap references non-existent config key: service-ca.crt Jul 08 19:46:13.335990 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.335960 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-f6w56"] Jul 08 19:46:13.339277 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:13.339240 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97eb1c0a_6d15_401b_b287_5e3fc9009fa0.slice/crio-54bea29749a58cbf83fbb3a9d014556efe1fd15747fa28c9a48ccd99055b2eaf WatchSource:0}: Error finding container 54bea29749a58cbf83fbb3a9d014556efe1fd15747fa28c9a48ccd99055b2eaf: Status 404 returned error can't find the container with id 54bea29749a58cbf83fbb3a9d014556efe1fd15747fa28c9a48ccd99055b2eaf Jul 08 19:46:13.593389 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.593305 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-f6w56" event={"ID":"97eb1c0a-6d15-401b-b287-5e3fc9009fa0","Type":"ContainerStarted","Data":"203691df8b8a953c6720a927a772581e202b9ad4343d75466839c480fb242950"} Jul 08 19:46:13.593389 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.593357 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-f6w56" event={"ID":"97eb1c0a-6d15-401b-b287-5e3fc9009fa0","Type":"ContainerStarted","Data":"54bea29749a58cbf83fbb3a9d014556efe1fd15747fa28c9a48ccd99055b2eaf"} Jul 08 19:46:13.609197 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.609135 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-f6w56" podStartSLOduration=1.6091189240000001 podStartE2EDuration="1.609118924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:13.608042698 +0000 UTC m=+244.772593103" watchObservedRunningTime="2026-07-08 19:46:13.609118924 +0000 UTC m=+244.773669327" Jul 08 19:46:13.894176 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:13.894079 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t7s4w_b75c8d82-d083-4cf2-9094-51739c38d0e5/node-ca/0.log" Jul 08 19:46:14.433261 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:14.433214 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:14.433469 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:14.433365 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:14.433545 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:14.433524 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:46:14.433603 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:14.433586 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls podName:6ea71571-304d-4d37-83fa-50bf418aab60 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:18.433569108 +0000 UTC m=+249.598119488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srckk" (UID: "6ea71571-304d-4d37-83fa-50bf418aab60") : secret "insights-runtime-extractor-tls" not found Jul 08 19:46:14.433969 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:14.433951 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:46:14.434036 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:14.434002 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm podName:6ea71571-304d-4d37-83fa-50bf418aab60 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:18.433988263 +0000 UTC m=+249.598538645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srckk" (UID: "6ea71571-304d-4d37-83fa-50bf418aab60") : configmap "kube-rbac-proxy" not found Jul 08 19:46:14.498898 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:14.498865 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fnphv_49fc0d84-1956-411d-b080-a249b75458f0/serve-healthcheck-canary/0.log" Jul 08 19:46:15.681427 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:15.681399 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:15.681805 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:15.681446 2572 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:15.681805 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:15.681759 2572 scope.go:117] "RemoveContainer" containerID="bb95337761170b63f90eceed664d3ca807bd21e65cd711bf9136d2832143d72d" Jul 08 19:46:15.681916 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:15.681898 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-l7rg9_openshift-console-operator(d2bf9f7a-9465-4c1c-8a14-4a9d39729d03)\"" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" podUID="d2bf9f7a-9465-4c1c-8a14-4a9d39729d03" Jul 08 19:46:18.467432 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:18.467399 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:18.467883 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:18.467451 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:18.467883 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:18.467528 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 19:46:18.467883 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:18.467579 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm podName:6ea71571-304d-4d37-83fa-50bf418aab60 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:26.467566202 +0000 UTC m=+257.632116583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm") pod "insights-runtime-extractor-srckk" (UID: "6ea71571-304d-4d37-83fa-50bf418aab60") : configmap "kube-rbac-proxy" not found Jul 08 19:46:18.467883 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:18.467582 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 19:46:18.467883 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:18.467650 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls podName:6ea71571-304d-4d37-83fa-50bf418aab60 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:26.467633034 +0000 UTC m=+257.632183423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls") pod "insights-runtime-extractor-srckk" (UID: "6ea71571-304d-4d37-83fa-50bf418aab60") : secret "insights-runtime-extractor-tls" not found Jul 08 19:46:20.288461 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:20.288415 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:46:20.288865 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:20.288585 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 19:46:20.288865 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:20.288653 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs podName:cc99e320-6751-4034-9446-70884569cdbb nodeName:}" failed. No retries permitted until 2026-07-08 19:48:22.288635934 +0000 UTC m=+373.453186324 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs") pod "network-metrics-daemon-pr9w2" (UID: "cc99e320-6751-4034-9446-70884569cdbb") : secret "metrics-daemon-secret" not found Jul 08 19:46:21.096704 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.096662 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:21.099117 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.099080 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e06a34ad-deaa-4b6e-9d0f-2da8d006f88f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nnmvk\" (UID: \"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:21.178599 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.178569 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-w78xq\"" Jul 08 19:46:21.183664 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.183628 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" Jul 08 19:46:21.298843 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.298804 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:21.299218 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.298915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:21.299603 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.299575 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43c4d634-0090-4a22-acd1-51deeaa3584e-service-ca-bundle\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:21.301583 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.301533 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43c4d634-0090-4a22-acd1-51deeaa3584e-metrics-certs\") pod \"router-default-74d89d68cf-sjvht\" (UID: \"43c4d634-0090-4a22-acd1-51deeaa3584e\") " pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:21.324141 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.324106 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk"] Jul 08 19:46:21.370424 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.370400 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4724n\"" Jul 08 19:46:21.378378 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.378349 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:21.508573 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.508543 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-74d89d68cf-sjvht"] Jul 08 19:46:21.512315 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:21.512291 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c4d634_0090_4a22_acd1_51deeaa3584e.slice/crio-39162a2600f85ba84b080416f3ad4e05d1953dd5a16cd184c71242e16064dabf WatchSource:0}: Error finding container 39162a2600f85ba84b080416f3ad4e05d1953dd5a16cd184c71242e16064dabf: Status 404 returned error can't find the container with id 39162a2600f85ba84b080416f3ad4e05d1953dd5a16cd184c71242e16064dabf Jul 08 19:46:21.619922 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.619827 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74d89d68cf-sjvht" event={"ID":"43c4d634-0090-4a22-acd1-51deeaa3584e","Type":"ContainerStarted","Data":"533756d75b17b1f4b9b78467d21ec0c4aa0e81a9e8014e8dd87004ae5f27c445"} Jul 08 19:46:21.619922 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.619881 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-74d89d68cf-sjvht" event={"ID":"43c4d634-0090-4a22-acd1-51deeaa3584e","Type":"ContainerStarted","Data":"39162a2600f85ba84b080416f3ad4e05d1953dd5a16cd184c71242e16064dabf"} Jul 08 19:46:21.620945 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.620923 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" event={"ID":"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f","Type":"ContainerStarted","Data":"0b070d55fb53ef2bda43d12d338cca104753e01d0fdf44ff209c9ea1c51fcbf8"} Jul 08 19:46:21.639123 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:21.639071 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-74d89d68cf-sjvht" podStartSLOduration=16.639055566 podStartE2EDuration="16.639055566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:21.637662052 +0000 UTC m=+252.802212455" watchObservedRunningTime="2026-07-08 19:46:21.639055566 +0000 UTC m=+252.803605963" Jul 08 19:46:22.379400 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:22.379362 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:22.381860 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:22.381835 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:22.625572 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:22.625540 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:22.626964 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:22.626939 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-74d89d68cf-sjvht" Jul 08 19:46:23.630885 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:23.630830 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" event={"ID":"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f","Type":"ContainerStarted","Data":"554be19689b2bcc84b3e55043f468899347765aa44341af3755390c888eb90aa"} Jul 08 19:46:23.631358 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:23.630946 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" event={"ID":"e06a34ad-deaa-4b6e-9d0f-2da8d006f88f","Type":"ContainerStarted","Data":"e8a462b76c9e73a5cf2f3afd6e0e9b7e9aaa2d00845786e735e55984210d83bb"} Jul 08 19:46:23.646311 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:23.646260 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nnmvk" podStartSLOduration=16.5239053 podStartE2EDuration="18.64624263s" totalImagesPullingTime="2.12233733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:23.645608253 +0000 UTC m=+254.810158679" watchObservedRunningTime="2026-07-08 19:46:23.64624263 +0000 UTC m=+254.810793070" Jul 08 19:46:26.548504 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:26.548471 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:26.548950 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:26.548518 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:26.549028 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:26.548955 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ea71571-304d-4d37-83fa-50bf418aab60-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:26.550897 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:26.550858 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ea71571-304d-4d37-83fa-50bf418aab60-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-srckk\" (UID: \"6ea71571-304d-4d37-83fa-50bf418aab60\") " pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:26.651832 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:26.651806 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-srckk" Jul 08 19:46:26.779285 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:26.779148 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-srckk"] Jul 08 19:46:26.781636 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:26.781610 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea71571_304d_4d37_83fa_50bf418aab60.slice/crio-48405d598bec55b08ea76471b7bd1326a922fa15cf41c39478bf1faade2610cc WatchSource:0}: Error finding container 48405d598bec55b08ea76471b7bd1326a922fa15cf41c39478bf1faade2610cc: Status 404 returned error can't find the container with id 48405d598bec55b08ea76471b7bd1326a922fa15cf41c39478bf1faade2610cc Jul 08 19:46:27.647843 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:27.647790 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srckk" event={"ID":"6ea71571-304d-4d37-83fa-50bf418aab60","Type":"ContainerStarted","Data":"48405d598bec55b08ea76471b7bd1326a922fa15cf41c39478bf1faade2610cc"} Jul 08 19:46:30.668384 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:30.668302 2572 scope.go:117] "RemoveContainer" containerID="bb95337761170b63f90eceed664d3ca807bd21e65cd711bf9136d2832143d72d" Jul 08 19:46:31.663399 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:31.663305 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srckk" event={"ID":"6ea71571-304d-4d37-83fa-50bf418aab60","Type":"ContainerStarted","Data":"6c6faac2c9ba19c79e43c2d6fcdd65b47277b415651bc207accf75a7cdcf7e86"} Jul 08 19:46:31.663399 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:31.663368 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srckk" event={"ID":"6ea71571-304d-4d37-83fa-50bf418aab60","Type":"ContainerStarted","Data":"d19c0251fbea47ffb700054590795c869dd665c44984574703c89f15c0edcfac"} Jul 08 19:46:31.664896 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:31.664877 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 19:46:31.665009 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:31.664927 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" event={"ID":"d2bf9f7a-9465-4c1c-8a14-4a9d39729d03","Type":"ContainerStarted","Data":"ea207e31d1e57b2f587ec93e667220ec6e0611e9b57195c38d8a002853204271"} Jul 08 19:46:31.665199 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:31.665182 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:31.671739 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:31.671721 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" Jul 08 19:46:31.680670 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:31.680629 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-l7rg9" podStartSLOduration=22.915536105 podStartE2EDuration="26.680617334s" totalImagesPullingTime="3.765081229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:31.679969372 +0000 UTC m=+262.844519775" watchObservedRunningTime="2026-07-08 19:46:31.680617334 +0000 UTC m=+262.845167764" Jul 08 19:46:33.443557 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.443520 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65785cfc6b-xgln2"] Jul 08 19:46:33.455037 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.455009 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.459302 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.459266 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 19:46:33.459527 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.459389 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4fdwr\"" Jul 08 19:46:33.459614 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.459554 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 19:46:33.460124 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.460103 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 19:46:33.466497 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.466466 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 19:46:33.467709 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.467688 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65785cfc6b-xgln2"] Jul 08 19:46:33.609610 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.609565 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxrxg\" (UniqueName: \"kubernetes.io/projected/9c72d281-8e1b-41fd-886a-14621c07949f-kube-api-access-nxrxg\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.609610 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.609617 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c72d281-8e1b-41fd-886a-14621c07949f-trusted-ca\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.609833 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.609718 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c72d281-8e1b-41fd-886a-14621c07949f-registry-certificates\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.609833 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.609754 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c72d281-8e1b-41fd-886a-14621c07949f-ca-trust-extracted\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.609833 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.609792 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c72d281-8e1b-41fd-886a-14621c07949f-installation-pull-secrets\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.610002 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.609836 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c72d281-8e1b-41fd-886a-14621c07949f-registry-tls\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.610002 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.609878 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c72d281-8e1b-41fd-886a-14621c07949f-image-registry-private-configuration\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.610002 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.609894 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c72d281-8e1b-41fd-886a-14621c07949f-bound-sa-token\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.711285 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.711140 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c72d281-8e1b-41fd-886a-14621c07949f-image-registry-private-configuration\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.711285 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.711208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c72d281-8e1b-41fd-886a-14621c07949f-bound-sa-token\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.711285 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.711260 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxrxg\" (UniqueName: \"kubernetes.io/projected/9c72d281-8e1b-41fd-886a-14621c07949f-kube-api-access-nxrxg\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.711559 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.711293 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c72d281-8e1b-41fd-886a-14621c07949f-trusted-ca\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.711559 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.711356 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c72d281-8e1b-41fd-886a-14621c07949f-registry-certificates\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.711559 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.711388 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c72d281-8e1b-41fd-886a-14621c07949f-ca-trust-extracted\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.711559 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.711418 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c72d281-8e1b-41fd-886a-14621c07949f-installation-pull-secrets\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.711559 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.711443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c72d281-8e1b-41fd-886a-14621c07949f-registry-tls\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.711949 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.711904 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c72d281-8e1b-41fd-886a-14621c07949f-ca-trust-extracted\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.712483 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.712424 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c72d281-8e1b-41fd-886a-14621c07949f-registry-certificates\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.712620 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.712599 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c72d281-8e1b-41fd-886a-14621c07949f-trusted-ca\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.714296 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.714272 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c72d281-8e1b-41fd-886a-14621c07949f-image-registry-private-configuration\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.714392 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.714293 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c72d281-8e1b-41fd-886a-14621c07949f-installation-pull-secrets\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.714586 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.714564 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c72d281-8e1b-41fd-886a-14621c07949f-registry-tls\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.719801 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.719778 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c72d281-8e1b-41fd-886a-14621c07949f-bound-sa-token\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.720372 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.720352 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxrxg\" (UniqueName: \"kubernetes.io/projected/9c72d281-8e1b-41fd-886a-14621c07949f-kube-api-access-nxrxg\") pod \"image-registry-65785cfc6b-xgln2\" (UID: \"9c72d281-8e1b-41fd-886a-14621c07949f\") " pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.766371 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.766335 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:33.965301 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:33.965233 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65785cfc6b-xgln2"] Jul 08 19:46:33.968876 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:33.968846 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c72d281_8e1b_41fd_886a_14621c07949f.slice/crio-8dcabe60a00324d99be5977457193035c6a3f5428b854ff584b73e7fc3a18cf8 WatchSource:0}: Error finding container 8dcabe60a00324d99be5977457193035c6a3f5428b854ff584b73e7fc3a18cf8: Status 404 returned error can't find the container with id 8dcabe60a00324d99be5977457193035c6a3f5428b854ff584b73e7fc3a18cf8 Jul 08 19:46:34.677644 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:34.677607 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-srckk" event={"ID":"6ea71571-304d-4d37-83fa-50bf418aab60","Type":"ContainerStarted","Data":"8834763d0a0309c956bf063ec38d1f735ec5d6bdbb735aabacb1e56102e398c2"} Jul 08 19:46:34.679078 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:34.679052 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" event={"ID":"9c72d281-8e1b-41fd-886a-14621c07949f","Type":"ContainerStarted","Data":"38fe8348b3742de1826bd8ff032ee68803d1ea7c565cb35e8ef8d843faebf892"} Jul 08 19:46:34.679196 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:34.679090 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" event={"ID":"9c72d281-8e1b-41fd-886a-14621c07949f","Type":"ContainerStarted","Data":"8dcabe60a00324d99be5977457193035c6a3f5428b854ff584b73e7fc3a18cf8"} Jul 08 19:46:34.679196 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:34.679108 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:34.693588 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:34.693539 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-srckk" podStartSLOduration=17.715833555 podStartE2EDuration="24.693525577s" totalImagesPullingTime="6.977692022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:46:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:34.692593067 +0000 UTC m=+265.857143469" watchObservedRunningTime="2026-07-08 19:46:34.693525577 +0000 UTC m=+265.858075979" Jul 08 19:46:34.709979 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:34.709916 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" podStartSLOduration=1.7099014110000001 podStartE2EDuration="1.709901411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:34.709321462 +0000 UTC m=+265.873871866" watchObservedRunningTime="2026-07-08 19:46:34.709901411 +0000 UTC m=+265.874451814" Jul 08 19:46:45.442974 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.442934 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dbfbddcfc-w6rgt"] Jul 08 19:46:45.687264 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.687231 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dbfbddcfc-w6rgt"] Jul 08 19:46:45.687435 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.687403 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.690789 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.690751 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 19:46:45.690789 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.690767 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 19:46:45.691003 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.690948 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 19:46:45.691003 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.690988 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 19:46:45.691117 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.691007 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 19:46:45.691117 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.690988 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 19:46:45.691117 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.691057 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 19:46:45.691117 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.690988 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nqqwm\"" Jul 08 19:46:45.695193 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.694859 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 19:46:45.703145 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.703125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-oauth-serving-cert\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.703261 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.703192 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-oauth-config\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.703261 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.703208 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-config\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.703261 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.703241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-trusted-ca-bundle\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.703261 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.703260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-service-ca\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.703412 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.703285 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np5fd\" (UniqueName: \"kubernetes.io/projected/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-kube-api-access-np5fd\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.703412 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.703362 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-serving-cert\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.804360 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.804326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-oauth-serving-cert\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.804540 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.804376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-oauth-config\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.804540 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.804398 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-config\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.804540 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.804432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-trusted-ca-bundle\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.804540 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.804449 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-service-ca\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.804540 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.804467 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np5fd\" (UniqueName: \"kubernetes.io/projected/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-kube-api-access-np5fd\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.804540 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.804512 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-serving-cert\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.805254 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.805230 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-config\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.805254 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.805246 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-oauth-serving-cert\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.805375 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.805288 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-service-ca\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.805413 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.805386 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-trusted-ca-bundle\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.806850 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.806831 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-oauth-config\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.807014 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.806998 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-serving-cert\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.812259 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.812241 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np5fd\" (UniqueName: \"kubernetes.io/projected/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-kube-api-access-np5fd\") pod \"console-dbfbddcfc-w6rgt\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:45.997404 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:45.997312 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:46.150955 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:46.150825 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dbfbddcfc-w6rgt"] Jul 08 19:46:46.716650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:46.716608 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dbfbddcfc-w6rgt" event={"ID":"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327","Type":"ContainerStarted","Data":"79de0e7f82ae7b74ed640bb3d4842502d04cf25ea3939e0374d85631cebaa3ee"} Jul 08 19:46:49.728941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:49.728903 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dbfbddcfc-w6rgt" event={"ID":"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327","Type":"ContainerStarted","Data":"bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23"} Jul 08 19:46:49.745389 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:49.745346 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dbfbddcfc-w6rgt" podStartSLOduration=1.7224769 podStartE2EDuration="4.745329712s" totalImagesPullingTime="3.022852812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:49.744559066 +0000 UTC m=+280.909109469" watchObservedRunningTime="2026-07-08 19:46:49.745329712 +0000 UTC m=+280.909880113" Jul 08 19:46:50.377980 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:50.377936 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-gg7bd" podUID="6d3f85dc-1bff-4fd1-940e-216b04a813ce" Jul 08 19:46:50.734255 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:50.734148 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gg7bd" Jul 08 19:46:53.573502 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:53.573464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:46:53.575705 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:53.575684 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d3f85dc-1bff-4fd1-940e-216b04a813ce-metrics-tls\") pod \"dns-default-gg7bd\" (UID: \"6d3f85dc-1bff-4fd1-940e-216b04a813ce\") " pod="openshift-dns/dns-default-gg7bd" Jul 08 19:46:53.737354 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:53.737322 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w5s5j\"" Jul 08 19:46:53.745899 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:53.745879 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gg7bd" Jul 08 19:46:53.770498 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:53.770461 2572 patch_prober.go:28] interesting pod/image-registry-65785cfc6b-xgln2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 19:46:53.770618 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:53.770519 2572 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" podUID="9c72d281-8e1b-41fd-886a-14621c07949f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 19:46:53.867792 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:53.867726 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gg7bd"] Jul 08 19:46:53.870587 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:53.870557 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3f85dc_1bff_4fd1_940e_216b04a813ce.slice/crio-24c794c61db4b6461a7182f0f0d81f66dd152303cc8d486397154fdfede0226b WatchSource:0}: Error finding container 24c794c61db4b6461a7182f0f0d81f66dd152303cc8d486397154fdfede0226b: Status 404 returned error can't find the container with id 24c794c61db4b6461a7182f0f0d81f66dd152303cc8d486397154fdfede0226b Jul 08 19:46:54.748744 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:54.748708 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gg7bd" event={"ID":"6d3f85dc-1bff-4fd1-940e-216b04a813ce","Type":"ContainerStarted","Data":"24c794c61db4b6461a7182f0f0d81f66dd152303cc8d486397154fdfede0226b"} Jul 08 19:46:55.686689 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:55.686617 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65785cfc6b-xgln2" Jul 08 19:46:55.754445 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:55.754410 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gg7bd" event={"ID":"6d3f85dc-1bff-4fd1-940e-216b04a813ce","Type":"ContainerStarted","Data":"a31a92337785862c4aab978cec67b655a431c44dc0ec99c3eda3501ff7f25bcd"} Jul 08 19:46:55.754874 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:55.754469 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gg7bd" event={"ID":"6d3f85dc-1bff-4fd1-940e-216b04a813ce","Type":"ContainerStarted","Data":"e35abb29e84ea289f26a589c192226b6c87434fbccec791ed5bf824cc092829b"} Jul 08 19:46:55.754874 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:55.754530 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gg7bd" Jul 08 19:46:55.775204 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:55.775130 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gg7bd" podStartSLOduration=251.304819594 podStartE2EDuration="4m12.775110987s" totalImagesPullingTime="1.470291393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:46:55.774102423 +0000 UTC m=+286.938652826" watchObservedRunningTime="2026-07-08 19:46:55.775110987 +0000 UTC m=+286.939661428" Jul 08 19:46:55.997588 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:55.997505 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:55.997588 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:55.997566 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:56.002272 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:56.002248 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:56.761413 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:56.761382 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:46:58.053657 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.053615 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf"] Jul 08 19:46:58.056247 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.056223 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.058387 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.058365 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 08 19:46:58.059097 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.059024 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 08 19:46:58.059097 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.059073 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kwb5p\"" Jul 08 19:46:58.059573 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.059554 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 19:46:58.059855 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.059835 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 19:46:58.059990 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.059674 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 19:46:58.071563 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.071541 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf"] Jul 08 19:46:58.074703 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.074682 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vdkf9"] Jul 08 19:46:58.077383 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.077366 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.079479 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.079462 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 19:46:58.079734 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.079716 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzqw7\"" Jul 08 19:46:58.079945 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.079928 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 19:46:58.081939 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.081920 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 19:46:58.113097 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113063 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f74755ad-e746-424c-94f4-55df332addd2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.113294 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113115 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f74755ad-e746-424c-94f4-55df332addd2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.113294 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113178 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jjr7\" (UniqueName: \"kubernetes.io/projected/f74755ad-e746-424c-94f4-55df332addd2-kube-api-access-9jjr7\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.113294 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113202 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-textfile\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.113294 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113245 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-tls\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.113294 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113268 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.113558 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113305 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-metrics-client-ca\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.113558 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113338 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-wtmp\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.113558 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113356 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-root\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.113558 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113380 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ck6x\" (UniqueName: \"kubernetes.io/projected/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-kube-api-access-6ck6x\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.113558 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113411 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-sys\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.113558 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113434 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.113558 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.113453 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f74755ad-e746-424c-94f4-55df332addd2-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.214225 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214188 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-wtmp\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.214225 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214225 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-root\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.214469 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214247 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ck6x\" (UniqueName: \"kubernetes.io/projected/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-kube-api-access-6ck6x\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.214469 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214271 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-sys\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.214469 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214331 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-sys\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.214469 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214368 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.214469 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214377 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-wtmp\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.214469 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214398 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f74755ad-e746-424c-94f4-55df332addd2-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.214469 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214442 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f74755ad-e746-424c-94f4-55df332addd2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.214803 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214509 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f74755ad-e746-424c-94f4-55df332addd2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.214803 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214547 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jjr7\" (UniqueName: \"kubernetes.io/projected/f74755ad-e746-424c-94f4-55df332addd2-kube-api-access-9jjr7\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.214803 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214570 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-textfile\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.214803 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214597 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-tls\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.214803 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214633 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.214803 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.214663 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-metrics-client-ca\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.215081 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.215026 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.215150 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.215130 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-metrics-client-ca\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.215315 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.215294 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-root\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.215438 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:58.215133 2572 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 08 19:46:58.215765 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:58.215248 2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 19:46:58.215765 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.215697 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-textfile\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.215765 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:58.215737 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-tls podName:1ca7b9e1-bc8d-4282-979e-99782cfcec6c nodeName:}" failed. No retries permitted until 2026-07-08 19:46:58.715717061 +0000 UTC m=+289.880267457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-tls") pod "node-exporter-vdkf9" (UID: "1ca7b9e1-bc8d-4282-979e-99782cfcec6c") : secret "node-exporter-tls" not found Jul 08 19:46:58.215984 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.215786 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f74755ad-e746-424c-94f4-55df332addd2-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.216100 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:46:58.216085 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f74755ad-e746-424c-94f4-55df332addd2-openshift-state-metrics-tls podName:f74755ad-e746-424c-94f4-55df332addd2 nodeName:}" failed. No retries permitted until 2026-07-08 19:46:58.716063107 +0000 UTC m=+289.880613528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f74755ad-e746-424c-94f4-55df332addd2-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-kq5rf" (UID: "f74755ad-e746-424c-94f4-55df332addd2") : secret "openshift-state-metrics-tls" not found Jul 08 19:46:58.218268 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.217945 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.218268 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.218196 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f74755ad-e746-424c-94f4-55df332addd2-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.226610 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.226588 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ck6x\" (UniqueName: \"kubernetes.io/projected/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-kube-api-access-6ck6x\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.227330 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.227307 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jjr7\" (UniqueName: \"kubernetes.io/projected/f74755ad-e746-424c-94f4-55df332addd2-kube-api-access-9jjr7\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.723611 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.723553 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-tls\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.723813 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.723658 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f74755ad-e746-424c-94f4-55df332addd2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.726099 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.726065 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1ca7b9e1-bc8d-4282-979e-99782cfcec6c-node-exporter-tls\") pod \"node-exporter-vdkf9\" (UID: \"1ca7b9e1-bc8d-4282-979e-99782cfcec6c\") " pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:58.726290 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.726271 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f74755ad-e746-424c-94f4-55df332addd2-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kq5rf\" (UID: \"f74755ad-e746-424c-94f4-55df332addd2\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.967446 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.967412 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" Jul 08 19:46:58.988380 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:58.988355 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vdkf9" Jul 08 19:46:59.004619 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:59.004571 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca7b9e1_bc8d_4282_979e_99782cfcec6c.slice/crio-6f4c41090e8ba44f0d5729d1aef0fcedc63781d44fc73cab833db7c7845b8ca5 WatchSource:0}: Error finding container 6f4c41090e8ba44f0d5729d1aef0fcedc63781d44fc73cab833db7c7845b8ca5: Status 404 returned error can't find the container with id 6f4c41090e8ba44f0d5729d1aef0fcedc63781d44fc73cab833db7c7845b8ca5 Jul 08 19:46:59.101495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:59.101458 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf"] Jul 08 19:46:59.105461 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:46:59.105434 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74755ad_e746_424c_94f4_55df332addd2.slice/crio-7f12b71f0c331ca3acb045fc6813f824e9a4254340c49ab6f6fb7b8558161b1b WatchSource:0}: Error finding container 7f12b71f0c331ca3acb045fc6813f824e9a4254340c49ab6f6fb7b8558161b1b: Status 404 returned error can't find the container with id 7f12b71f0c331ca3acb045fc6813f824e9a4254340c49ab6f6fb7b8558161b1b Jul 08 19:46:59.769367 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:59.769328 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdkf9" event={"ID":"1ca7b9e1-bc8d-4282-979e-99782cfcec6c","Type":"ContainerStarted","Data":"6f4c41090e8ba44f0d5729d1aef0fcedc63781d44fc73cab833db7c7845b8ca5"} Jul 08 19:46:59.771049 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:59.771019 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" event={"ID":"f74755ad-e746-424c-94f4-55df332addd2","Type":"ContainerStarted","Data":"ab883c54d44a211a70387caddc57bf27ba5ee26c095eebeb63c65405fc3285ec"} Jul 08 19:46:59.771181 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:59.771067 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" event={"ID":"f74755ad-e746-424c-94f4-55df332addd2","Type":"ContainerStarted","Data":"04d8f9492ff74c83149a49c770ea87f2f90ab19731a5e39c6f13faa4f9ecba96"} Jul 08 19:46:59.771181 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:46:59.771088 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" event={"ID":"f74755ad-e746-424c-94f4-55df332addd2","Type":"ContainerStarted","Data":"7f12b71f0c331ca3acb045fc6813f824e9a4254340c49ab6f6fb7b8558161b1b"} Jul 08 19:47:00.776440 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:00.776403 2572 generic.go:356] "Generic (PLEG): container finished" podID="1ca7b9e1-bc8d-4282-979e-99782cfcec6c" containerID="765d72dbcf134ea8178c39902e6d8169f761fd58e95915999667b44af155b664" exitCode=0 Jul 08 19:47:00.776918 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:00.776494 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdkf9" event={"ID":"1ca7b9e1-bc8d-4282-979e-99782cfcec6c","Type":"ContainerDied","Data":"765d72dbcf134ea8178c39902e6d8169f761fd58e95915999667b44af155b664"} Jul 08 19:47:00.778471 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:00.778447 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" event={"ID":"f74755ad-e746-424c-94f4-55df332addd2","Type":"ContainerStarted","Data":"8dc4d62879a02e826243cef4d4f2239202204b4488352fa84cc5c6723e5fd58e"} Jul 08 19:47:00.810934 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:00.810878 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kq5rf" podStartSLOduration=1.733927789 podStartE2EDuration="2.810860718s" totalImagesPullingTime="1.076932929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:47:00.809576654 +0000 UTC m=+291.974127057" watchObservedRunningTime="2026-07-08 19:47:00.810860718 +0000 UTC m=+291.975411121" Jul 08 19:47:01.783729 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:01.783688 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdkf9" event={"ID":"1ca7b9e1-bc8d-4282-979e-99782cfcec6c","Type":"ContainerStarted","Data":"d20ce6ab2f727968ac6ce0beef22e7c612669eef27878b9c2fdfb5bc22fef610"} Jul 08 19:47:01.784104 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:01.783754 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vdkf9" event={"ID":"1ca7b9e1-bc8d-4282-979e-99782cfcec6c","Type":"ContainerStarted","Data":"62dd0a7c3bb4413806046021292cba1da3435520f0cb0154fe33cb71c9554268"} Jul 08 19:47:01.801832 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:01.801791 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vdkf9" podStartSLOduration=2.957666703 podStartE2EDuration="3.80177671s" totalImagesPullingTime="773.026297ms" totalInitContainerRuntime="71.08371ms" isStatefulPod=true podCreationTimestamp="2026-07-08 19:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:47:01.800517228 +0000 UTC m=+292.965067631" watchObservedRunningTime="2026-07-08 19:47:01.80177671 +0000 UTC m=+292.966327126" Jul 08 19:47:02.669841 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:47:02.669789 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-pr9w2" podUID="cc99e320-6751-4034-9446-70884569cdbb" Jul 08 19:47:05.760379 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:05.760350 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gg7bd" Jul 08 19:47:09.444642 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:09.444600 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 19:47:09.445130 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:09.444838 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 19:47:09.459292 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:09.459262 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 19:47:13.667890 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:13.667862 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:47:20.851962 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:20.851921 2572 generic.go:356] "Generic (PLEG): container finished" podID="b68237ef-7e03-4ded-a8e2-3e69518b6736" containerID="49f6441a4f3a65085c4c11a55d2e7361b71d00b51243d90aaf47d4b5d80e7a35" exitCode=0 Jul 08 19:47:20.852501 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:20.851985 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" event={"ID":"b68237ef-7e03-4ded-a8e2-3e69518b6736","Type":"ContainerDied","Data":"49f6441a4f3a65085c4c11a55d2e7361b71d00b51243d90aaf47d4b5d80e7a35"} Jul 08 19:47:20.852501 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:20.852356 2572 scope.go:117] "RemoveContainer" containerID="49f6441a4f3a65085c4c11a55d2e7361b71d00b51243d90aaf47d4b5d80e7a35" Jul 08 19:47:21.857739 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:21.857706 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ll9pj" event={"ID":"b68237ef-7e03-4ded-a8e2-3e69518b6736","Type":"ContainerStarted","Data":"11baa76c4862592905dfd0be5695865ebbc11b09060c0cf08894e66364914d2d"} Jul 08 19:47:21.982634 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:21.982597 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dbfbddcfc-w6rgt"] Jul 08 19:47:35.907896 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:35.907859 2572 generic.go:356] "Generic (PLEG): container finished" podID="4ea05414-ca40-4d7e-a66d-40dd97d8a2a7" containerID="063ab80aacb503738395b8786c308685aaf70a14ddc0c8eb683b13c61154474c" exitCode=0 Jul 08 19:47:35.908344 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:35.907937 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" event={"ID":"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7","Type":"ContainerDied","Data":"063ab80aacb503738395b8786c308685aaf70a14ddc0c8eb683b13c61154474c"} Jul 08 19:47:35.908344 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:35.908318 2572 scope.go:117] "RemoveContainer" containerID="063ab80aacb503738395b8786c308685aaf70a14ddc0c8eb683b13c61154474c" Jul 08 19:47:36.912952 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:36.912918 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-b8wqw" event={"ID":"4ea05414-ca40-4d7e-a66d-40dd97d8a2a7","Type":"ContainerStarted","Data":"3518b661b3d3429ad7cd228364d4eb4f50da89473af924dbe3d153127ae9ff17"} Jul 08 19:47:45.949140 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:45.949103 2572 generic.go:356] "Generic (PLEG): container finished" podID="2e4617b8-a240-411a-a380-0408e88ed3f1" containerID="7c18e65089dc0f4e1b47fc99af45b9d0fce7e44fc8fbb700160fc345637489cc" exitCode=0 Jul 08 19:47:45.949554 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:45.949179 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" event={"ID":"2e4617b8-a240-411a-a380-0408e88ed3f1","Type":"ContainerDied","Data":"7c18e65089dc0f4e1b47fc99af45b9d0fce7e44fc8fbb700160fc345637489cc"} Jul 08 19:47:45.949554 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:45.949505 2572 scope.go:117] "RemoveContainer" containerID="7c18e65089dc0f4e1b47fc99af45b9d0fce7e44fc8fbb700160fc345637489cc" Jul 08 19:47:46.954745 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:46.954711 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-x4b9z" event={"ID":"2e4617b8-a240-411a-a380-0408e88ed3f1","Type":"ContainerStarted","Data":"de1afe615fab53f37e064ec51891d502defeee0c2fc7281aee65d2ac43d7c40f"} Jul 08 19:47:47.005540 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.005500 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-dbfbddcfc-w6rgt" podUID="efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" containerName="console" containerID="cri-o://bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23" gracePeriod=15 Jul 08 19:47:47.252952 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.252927 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:47:47.345471 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.345440 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-service-ca\") pod \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " Jul 08 19:47:47.345590 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.345486 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-config\") pod \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " Jul 08 19:47:47.345590 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.345510 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-oauth-serving-cert\") pod \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " Jul 08 19:47:47.345590 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.345539 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-serving-cert\") pod \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " Jul 08 19:47:47.345687 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.345652 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-trusted-ca-bundle\") pod \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " Jul 08 19:47:47.345735 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.345686 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-oauth-config\") pod \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " Jul 08 19:47:47.345735 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.345714 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np5fd\" (UniqueName: \"kubernetes.io/projected/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-kube-api-access-np5fd\") pod \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\" (UID: \"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327\") " Jul 08 19:47:47.345958 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.345871 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-service-ca" (OuterVolumeSpecName: "service-ca") pod "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" (UID: "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:47:47.345958 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.345920 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" (UID: "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:47:47.345958 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.345925 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-config" (OuterVolumeSpecName: "console-config") pod "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" (UID: "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:47:47.346212 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.346031 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:47:47.346212 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.346044 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-config\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:47:47.346212 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.346052 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-oauth-serving-cert\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:47:47.346212 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.346082 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" (UID: "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:47:47.347882 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.347851 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" (UID: "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:47:47.347982 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.347884 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-kube-api-access-np5fd" (OuterVolumeSpecName: "kube-api-access-np5fd") pod "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" (UID: "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327"). InnerVolumeSpecName "kube-api-access-np5fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:47:47.347982 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.347890 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" (UID: "efc7bc77-fff8-4cdf-9c3d-841aa5a3d327"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:47:47.446514 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.446473 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-serving-cert\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:47:47.446514 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.446503 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-trusted-ca-bundle\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:47:47.446514 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.446512 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-console-oauth-config\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:47:47.446514 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.446523 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np5fd\" (UniqueName: \"kubernetes.io/projected/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327-kube-api-access-np5fd\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:47:47.959852 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.959763 2572 generic.go:356] "Generic (PLEG): container finished" podID="efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" containerID="bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23" exitCode=0 Jul 08 19:47:47.959852 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.959807 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dbfbddcfc-w6rgt" Jul 08 19:47:47.960332 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.959849 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dbfbddcfc-w6rgt" event={"ID":"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327","Type":"ContainerDied","Data":"bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23"} Jul 08 19:47:47.960332 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.959903 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dbfbddcfc-w6rgt" event={"ID":"efc7bc77-fff8-4cdf-9c3d-841aa5a3d327","Type":"ContainerDied","Data":"79de0e7f82ae7b74ed640bb3d4842502d04cf25ea3939e0374d85631cebaa3ee"} Jul 08 19:47:47.960332 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.959929 2572 scope.go:117] "RemoveContainer" containerID="bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23" Jul 08 19:47:47.968056 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.968035 2572 scope.go:117] "RemoveContainer" containerID="bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23" Jul 08 19:47:47.968415 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:47:47.968393 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23\": container with ID starting with bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23 not found: ID does not exist" containerID="bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23" Jul 08 19:47:47.968492 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.968429 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23"} err="failed to get container status \"bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23\": rpc error: code = NotFound desc = could not find container \"bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23\": container with ID starting with bd3510ad278614221289ee4caf8db062d402ff25bcdaab5bc93efd4725e96c23 not found: ID does not exist" Jul 08 19:47:47.975322 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.975296 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dbfbddcfc-w6rgt"] Jul 08 19:47:47.979340 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:47.979319 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dbfbddcfc-w6rgt"] Jul 08 19:47:49.672294 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:47:49.672261 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" path="/var/lib/kubelet/pods/efc7bc77-fff8-4cdf-9c3d-841aa5a3d327/volumes" Jul 08 19:48:22.337078 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:22.337037 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:48:22.339462 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:22.339433 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc99e320-6751-4034-9446-70884569cdbb-metrics-certs\") pod \"network-metrics-daemon-pr9w2\" (UID: \"cc99e320-6751-4034-9446-70884569cdbb\") " pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:48:22.370589 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:22.370559 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dpn4j\"" Jul 08 19:48:22.379420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:22.379393 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pr9w2" Jul 08 19:48:22.508648 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:22.508613 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pr9w2"] Jul 08 19:48:22.511965 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:48:22.511927 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc99e320_6751_4034_9446_70884569cdbb.slice/crio-bb9c5033e79ece2296604f57785f46ecba4d63cd8241a91d8a6a775dcc080ca5 WatchSource:0}: Error finding container bb9c5033e79ece2296604f57785f46ecba4d63cd8241a91d8a6a775dcc080ca5: Status 404 returned error can't find the container with id bb9c5033e79ece2296604f57785f46ecba4d63cd8241a91d8a6a775dcc080ca5 Jul 08 19:48:22.513872 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:22.513855 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:48:23.086350 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.086311 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pr9w2" event={"ID":"cc99e320-6751-4034-9446-70884569cdbb","Type":"ContainerStarted","Data":"bb9c5033e79ece2296604f57785f46ecba4d63cd8241a91d8a6a775dcc080ca5"} Jul 08 19:48:23.463540 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.463450 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9bd8bbd4-lzmgh"] Jul 08 19:48:23.463993 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.463850 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" containerName="console" Jul 08 19:48:23.463993 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.463868 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" containerName="console" Jul 08 19:48:23.463993 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.463952 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc7bc77-fff8-4cdf-9c3d-841aa5a3d327" containerName="console" Jul 08 19:48:23.467187 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.467149 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.469386 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.469361 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 19:48:23.469492 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.469370 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nqqwm\"" Jul 08 19:48:23.470388 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.470119 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 19:48:23.470388 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.470149 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 19:48:23.470388 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.470181 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 19:48:23.470388 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.470231 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 19:48:23.470388 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.470312 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 19:48:23.470388 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.470381 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 19:48:23.475752 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.475698 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 19:48:23.479736 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.479709 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9bd8bbd4-lzmgh"] Jul 08 19:48:23.549232 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.549200 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-trusted-ca-bundle\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.549399 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.549236 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mwbl\" (UniqueName: \"kubernetes.io/projected/b838c65f-309a-4939-92c2-b0a10b0b1273-kube-api-access-5mwbl\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.549399 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.549284 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-oauth-config\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.549399 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.549350 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-console-config\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.549399 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.549389 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-service-ca\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.549589 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.549434 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-oauth-serving-cert\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.549589 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.549477 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-serving-cert\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.650348 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.649809 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-oauth-serving-cert\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.650348 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.649863 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-serving-cert\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.650348 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.649900 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-trusted-ca-bundle\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.650348 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.649925 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mwbl\" (UniqueName: \"kubernetes.io/projected/b838c65f-309a-4939-92c2-b0a10b0b1273-kube-api-access-5mwbl\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.650348 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.649998 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-oauth-config\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.650348 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.650026 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-console-config\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.650348 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.650055 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-service-ca\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.651136 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.650770 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-service-ca\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.651136 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.650836 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-trusted-ca-bundle\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.651136 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.650651 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-oauth-serving-cert\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.651425 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.651404 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-console-config\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.652937 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.652915 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-oauth-config\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.653012 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.652972 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-serving-cert\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.659364 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.659338 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mwbl\" (UniqueName: \"kubernetes.io/projected/b838c65f-309a-4939-92c2-b0a10b0b1273-kube-api-access-5mwbl\") pod \"console-7c9bd8bbd4-lzmgh\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.779301 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.779255 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:23.914423 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:23.914388 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9bd8bbd4-lzmgh"] Jul 08 19:48:23.918386 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:48:23.918360 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb838c65f_309a_4939_92c2_b0a10b0b1273.slice/crio-861d88c0d7e050f23c4c41ab9c8625cd3f8a9efa184625210eda1273a927d4ea WatchSource:0}: Error finding container 861d88c0d7e050f23c4c41ab9c8625cd3f8a9efa184625210eda1273a927d4ea: Status 404 returned error can't find the container with id 861d88c0d7e050f23c4c41ab9c8625cd3f8a9efa184625210eda1273a927d4ea Jul 08 19:48:24.091956 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:24.091848 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd8bbd4-lzmgh" event={"ID":"b838c65f-309a-4939-92c2-b0a10b0b1273","Type":"ContainerStarted","Data":"c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1"} Jul 08 19:48:24.091956 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:24.091915 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd8bbd4-lzmgh" event={"ID":"b838c65f-309a-4939-92c2-b0a10b0b1273","Type":"ContainerStarted","Data":"861d88c0d7e050f23c4c41ab9c8625cd3f8a9efa184625210eda1273a927d4ea"} Jul 08 19:48:24.093530 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:24.093504 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pr9w2" event={"ID":"cc99e320-6751-4034-9446-70884569cdbb","Type":"ContainerStarted","Data":"cafb1f8d93bcb180969e638be90237b104c150378c9bf2357d0b80d41cd4c033"} Jul 08 19:48:24.093684 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:24.093558 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pr9w2" event={"ID":"cc99e320-6751-4034-9446-70884569cdbb","Type":"ContainerStarted","Data":"bccd0dc57a99afc95feda5a581742a46fb01ffe9b091143c298389047b478f3b"} Jul 08 19:48:24.108502 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:24.108444 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c9bd8bbd4-lzmgh" podStartSLOduration=1.108426314 podStartE2EDuration="1.108426314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:48:24.107492746 +0000 UTC m=+375.272043150" watchObservedRunningTime="2026-07-08 19:48:24.108426314 +0000 UTC m=+375.272976718" Jul 08 19:48:24.121807 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:24.121751 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pr9w2" podStartSLOduration=374.169095167 podStartE2EDuration="6m15.121734681s" totalImagesPullingTime="952.639514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:42:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:48:24.120853433 +0000 UTC m=+375.285403836" watchObservedRunningTime="2026-07-08 19:48:24.121734681 +0000 UTC m=+375.286285098" Jul 08 19:48:33.780183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:33.780124 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:33.780183 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:33.780185 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:33.784781 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:33.784757 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:48:34.135613 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:48:34.135539 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:49:04.536526 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.536494 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn"] Jul 08 19:49:04.539708 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.539692 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.542125 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.542104 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 19:49:04.542258 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.542107 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 19:49:04.542919 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.542905 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s7xwf\"" Jul 08 19:49:04.549367 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.549348 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn"] Jul 08 19:49:04.627941 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.627910 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.628112 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.627959 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.628112 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.627985 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4fg\" (UniqueName: \"kubernetes.io/projected/3b4b388e-d2a0-4010-ba12-4600b6fc159d-kube-api-access-zt4fg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.728693 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.728666 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.728788 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.728711 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.728788 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.728737 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4fg\" (UniqueName: \"kubernetes.io/projected/3b4b388e-d2a0-4010-ba12-4600b6fc159d-kube-api-access-zt4fg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.729128 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.729105 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.729189 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.729117 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.736990 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.736965 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4fg\" (UniqueName: \"kubernetes.io/projected/3b4b388e-d2a0-4010-ba12-4600b6fc159d-kube-api-access-zt4fg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.849736 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.849673 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:04.975187 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:04.975084 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn"] Jul 08 19:49:04.977882 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:49:04.977851 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4b388e_d2a0_4010_ba12_4600b6fc159d.slice/crio-e6c8443bb8d5bf8c39aeab8d2b7220625c324250fbe6f04eb9f724865b4ccf2e WatchSource:0}: Error finding container e6c8443bb8d5bf8c39aeab8d2b7220625c324250fbe6f04eb9f724865b4ccf2e: Status 404 returned error can't find the container with id e6c8443bb8d5bf8c39aeab8d2b7220625c324250fbe6f04eb9f724865b4ccf2e Jul 08 19:49:05.246226 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:05.246135 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" event={"ID":"3b4b388e-d2a0-4010-ba12-4600b6fc159d","Type":"ContainerStarted","Data":"e6c8443bb8d5bf8c39aeab8d2b7220625c324250fbe6f04eb9f724865b4ccf2e"} Jul 08 19:49:13.277966 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:13.277926 2572 generic.go:356] "Generic (PLEG): container finished" podID="3b4b388e-d2a0-4010-ba12-4600b6fc159d" containerID="4f8da4e28fa20c3f2a6ca430e930db87059492c400ea7379695b9d27f077e813" exitCode=0 Jul 08 19:49:13.278475 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:13.277996 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" event={"ID":"3b4b388e-d2a0-4010-ba12-4600b6fc159d","Type":"ContainerDied","Data":"4f8da4e28fa20c3f2a6ca430e930db87059492c400ea7379695b9d27f077e813"} Jul 08 19:49:18.304946 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:18.304912 2572 generic.go:356] "Generic (PLEG): container finished" podID="3b4b388e-d2a0-4010-ba12-4600b6fc159d" containerID="5e9af4f23bb6cb976feecb5fcae234ad2852e2b660e44b9bd9a025de424763f4" exitCode=0 Jul 08 19:49:18.305337 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:18.304973 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" event={"ID":"3b4b388e-d2a0-4010-ba12-4600b6fc159d","Type":"ContainerDied","Data":"5e9af4f23bb6cb976feecb5fcae234ad2852e2b660e44b9bd9a025de424763f4"} Jul 08 19:49:25.335325 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:25.335295 2572 generic.go:356] "Generic (PLEG): container finished" podID="3b4b388e-d2a0-4010-ba12-4600b6fc159d" containerID="094577ca106dc85ed38ff510384d0e8eacdee94d84f85cee268ac2529e7e9b18" exitCode=0 Jul 08 19:49:25.335665 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:25.335380 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" event={"ID":"3b4b388e-d2a0-4010-ba12-4600b6fc159d","Type":"ContainerDied","Data":"094577ca106dc85ed38ff510384d0e8eacdee94d84f85cee268ac2529e7e9b18"} Jul 08 19:49:26.501207 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:26.501178 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:26.598464 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:26.598403 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-bundle\") pod \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " Jul 08 19:49:26.598565 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:26.598466 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt4fg\" (UniqueName: \"kubernetes.io/projected/3b4b388e-d2a0-4010-ba12-4600b6fc159d-kube-api-access-zt4fg\") pod \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " Jul 08 19:49:26.598565 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:26.598495 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-util\") pod \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\" (UID: \"3b4b388e-d2a0-4010-ba12-4600b6fc159d\") " Jul 08 19:49:26.599092 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:26.599070 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-bundle" (OuterVolumeSpecName: "bundle") pod "3b4b388e-d2a0-4010-ba12-4600b6fc159d" (UID: "3b4b388e-d2a0-4010-ba12-4600b6fc159d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:49:26.600548 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:26.600518 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4b388e-d2a0-4010-ba12-4600b6fc159d-kube-api-access-zt4fg" (OuterVolumeSpecName: "kube-api-access-zt4fg") pod "3b4b388e-d2a0-4010-ba12-4600b6fc159d" (UID: "3b4b388e-d2a0-4010-ba12-4600b6fc159d"). InnerVolumeSpecName "kube-api-access-zt4fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:49:26.602985 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:26.602964 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-util" (OuterVolumeSpecName: "util") pod "3b4b388e-d2a0-4010-ba12-4600b6fc159d" (UID: "3b4b388e-d2a0-4010-ba12-4600b6fc159d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:49:26.699224 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:26.699187 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-bundle\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:49:26.699224 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:26.699220 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt4fg\" (UniqueName: \"kubernetes.io/projected/3b4b388e-d2a0-4010-ba12-4600b6fc159d-kube-api-access-zt4fg\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:49:26.699224 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:26.699229 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b4b388e-d2a0-4010-ba12-4600b6fc159d-util\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:49:27.344312 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:27.344275 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" event={"ID":"3b4b388e-d2a0-4010-ba12-4600b6fc159d","Type":"ContainerDied","Data":"e6c8443bb8d5bf8c39aeab8d2b7220625c324250fbe6f04eb9f724865b4ccf2e"} Jul 08 19:49:27.344475 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:27.344328 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6c8443bb8d5bf8c39aeab8d2b7220625c324250fbe6f04eb9f724865b4ccf2e" Jul 08 19:49:27.344475 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:27.344285 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540vc4dn" Jul 08 19:49:32.591032 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.590959 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk"] Jul 08 19:49:32.591395 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.591259 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4b388e-d2a0-4010-ba12-4600b6fc159d" containerName="extract" Jul 08 19:49:32.591395 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.591271 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4b388e-d2a0-4010-ba12-4600b6fc159d" containerName="extract" Jul 08 19:49:32.591395 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.591292 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4b388e-d2a0-4010-ba12-4600b6fc159d" containerName="pull" Jul 08 19:49:32.591395 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.591301 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4b388e-d2a0-4010-ba12-4600b6fc159d" containerName="pull" Jul 08 19:49:32.591395 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.591310 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4b388e-d2a0-4010-ba12-4600b6fc159d" containerName="util" Jul 08 19:49:32.591395 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.591315 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4b388e-d2a0-4010-ba12-4600b6fc159d" containerName="util" Jul 08 19:49:32.591395 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.591359 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b4b388e-d2a0-4010-ba12-4600b6fc159d" containerName="extract" Jul 08 19:49:32.593266 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.593251 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" Jul 08 19:49:32.595542 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.595520 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 08 19:49:32.595831 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.595807 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:49:32.595945 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.595850 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-djvwq\"" Jul 08 19:49:32.608775 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.608754 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk"] Jul 08 19:49:32.638909 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.638883 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwcp\" (UniqueName: \"kubernetes.io/projected/7bc44236-da60-41e6-a7c8-a0b09eb38419-kube-api-access-gxwcp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t8nbk\" (UID: \"7bc44236-da60-41e6-a7c8-a0b09eb38419\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" Jul 08 19:49:32.639017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.638923 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bc44236-da60-41e6-a7c8-a0b09eb38419-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t8nbk\" (UID: \"7bc44236-da60-41e6-a7c8-a0b09eb38419\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" Jul 08 19:49:32.739892 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.739856 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bc44236-da60-41e6-a7c8-a0b09eb38419-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t8nbk\" (UID: \"7bc44236-da60-41e6-a7c8-a0b09eb38419\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" Jul 08 19:49:32.739992 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.739923 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwcp\" (UniqueName: \"kubernetes.io/projected/7bc44236-da60-41e6-a7c8-a0b09eb38419-kube-api-access-gxwcp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t8nbk\" (UID: \"7bc44236-da60-41e6-a7c8-a0b09eb38419\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" Jul 08 19:49:32.740226 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.740211 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7bc44236-da60-41e6-a7c8-a0b09eb38419-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t8nbk\" (UID: \"7bc44236-da60-41e6-a7c8-a0b09eb38419\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" Jul 08 19:49:32.751206 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.751179 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwcp\" (UniqueName: \"kubernetes.io/projected/7bc44236-da60-41e6-a7c8-a0b09eb38419-kube-api-access-gxwcp\") pod \"cert-manager-operator-controller-manager-55c444bf86-t8nbk\" (UID: \"7bc44236-da60-41e6-a7c8-a0b09eb38419\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" Jul 08 19:49:32.902202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:32.902138 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" Jul 08 19:49:33.053328 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:33.053254 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk"] Jul 08 19:49:33.055649 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:49:33.055621 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc44236_da60_41e6_a7c8_a0b09eb38419.slice/crio-c7f912c675a1ed54a0ba86cc54fc809b6dc5040a0653fc769b7df2ecfc98a22b WatchSource:0}: Error finding container c7f912c675a1ed54a0ba86cc54fc809b6dc5040a0653fc769b7df2ecfc98a22b: Status 404 returned error can't find the container with id c7f912c675a1ed54a0ba86cc54fc809b6dc5040a0653fc769b7df2ecfc98a22b Jul 08 19:49:33.367335 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:33.367305 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" event={"ID":"7bc44236-da60-41e6-a7c8-a0b09eb38419","Type":"ContainerStarted","Data":"c7f912c675a1ed54a0ba86cc54fc809b6dc5040a0653fc769b7df2ecfc98a22b"} Jul 08 19:49:42.404559 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:42.404523 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" event={"ID":"7bc44236-da60-41e6-a7c8-a0b09eb38419","Type":"ContainerStarted","Data":"0b49fd8a5cf535f40ea615d6718274d004a5aef0356c517417bcdeb6aeabaae1"} Jul 08 19:49:42.426784 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:42.426741 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-t8nbk" podStartSLOduration=1.927011844 podStartE2EDuration="10.426726178s" totalImagesPullingTime="8.499714334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:49:42.424573238 +0000 UTC m=+453.589123640" watchObservedRunningTime="2026-07-08 19:49:42.426726178 +0000 UTC m=+453.591276579" Jul 08 19:49:43.405748 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.405712 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xg8vv"] Jul 08 19:49:43.408251 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.408235 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" Jul 08 19:49:43.410445 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.410422 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6p6dt\"" Jul 08 19:49:43.410445 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.410429 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 19:49:43.411177 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.411145 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 19:49:43.416116 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.416097 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xg8vv"] Jul 08 19:49:43.511291 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.511266 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfc9b264-2348-476e-bc67-7368923e2087-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xg8vv\" (UID: \"cfc9b264-2348-476e-bc67-7368923e2087\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" Jul 08 19:49:43.511400 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.511298 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhr9s\" (UniqueName: \"kubernetes.io/projected/cfc9b264-2348-476e-bc67-7368923e2087-kube-api-access-zhr9s\") pod \"cert-manager-webhook-6486ffd997-xg8vv\" (UID: \"cfc9b264-2348-476e-bc67-7368923e2087\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" Jul 08 19:49:43.612522 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.612497 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfc9b264-2348-476e-bc67-7368923e2087-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xg8vv\" (UID: \"cfc9b264-2348-476e-bc67-7368923e2087\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" Jul 08 19:49:43.612634 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.612537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhr9s\" (UniqueName: \"kubernetes.io/projected/cfc9b264-2348-476e-bc67-7368923e2087-kube-api-access-zhr9s\") pod \"cert-manager-webhook-6486ffd997-xg8vv\" (UID: \"cfc9b264-2348-476e-bc67-7368923e2087\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" Jul 08 19:49:43.623288 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.623264 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cfc9b264-2348-476e-bc67-7368923e2087-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xg8vv\" (UID: \"cfc9b264-2348-476e-bc67-7368923e2087\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" Jul 08 19:49:43.623550 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.623530 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhr9s\" (UniqueName: \"kubernetes.io/projected/cfc9b264-2348-476e-bc67-7368923e2087-kube-api-access-zhr9s\") pod \"cert-manager-webhook-6486ffd997-xg8vv\" (UID: \"cfc9b264-2348-476e-bc67-7368923e2087\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" Jul 08 19:49:43.728719 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.728658 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" Jul 08 19:49:43.863413 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:43.863390 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xg8vv"] Jul 08 19:49:43.865660 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:49:43.865635 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfc9b264_2348_476e_bc67_7368923e2087.slice/crio-5b20249df07a734c1d1cda537bb07ea4d0e34fcc4ce0e402a90e5ce598f68756 WatchSource:0}: Error finding container 5b20249df07a734c1d1cda537bb07ea4d0e34fcc4ce0e402a90e5ce598f68756: Status 404 returned error can't find the container with id 5b20249df07a734c1d1cda537bb07ea4d0e34fcc4ce0e402a90e5ce598f68756 Jul 08 19:49:44.413122 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.413089 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" event={"ID":"cfc9b264-2348-476e-bc67-7368923e2087","Type":"ContainerStarted","Data":"5b20249df07a734c1d1cda537bb07ea4d0e34fcc4ce0e402a90e5ce598f68756"} Jul 08 19:49:44.613409 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.613379 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r"] Jul 08 19:49:44.616542 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.616528 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:44.618600 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.618579 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s7xwf\"" Jul 08 19:49:44.618692 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.618630 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 19:49:44.619136 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.619123 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 19:49:44.623933 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.623911 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r"] Jul 08 19:49:44.721093 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.721021 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:44.721220 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.721096 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lnvj\" (UniqueName: \"kubernetes.io/projected/bfafd352-d5de-4b2b-94d7-4a865c9512a7-kube-api-access-9lnvj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:44.721220 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.721128 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:44.821698 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.821665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:44.821850 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.821833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lnvj\" (UniqueName: \"kubernetes.io/projected/bfafd352-d5de-4b2b-94d7-4a865c9512a7-kube-api-access-9lnvj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:44.821919 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.821860 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:44.822222 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.822196 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:44.822343 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.822198 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:44.829604 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.829580 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lnvj\" (UniqueName: \"kubernetes.io/projected/bfafd352-d5de-4b2b-94d7-4a865c9512a7-kube-api-access-9lnvj\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:44.926041 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:44.926016 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:45.051922 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:45.051828 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r"] Jul 08 19:49:45.054518 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:49:45.054489 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfafd352_d5de_4b2b_94d7_4a865c9512a7.slice/crio-245107f4b89a0473c45c47a3417f3cb8eaeb8f475f66cdc4bdc3e27f417cbe55 WatchSource:0}: Error finding container 245107f4b89a0473c45c47a3417f3cb8eaeb8f475f66cdc4bdc3e27f417cbe55: Status 404 returned error can't find the container with id 245107f4b89a0473c45c47a3417f3cb8eaeb8f475f66cdc4bdc3e27f417cbe55 Jul 08 19:49:45.419349 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:45.419305 2572 generic.go:356] "Generic (PLEG): container finished" podID="bfafd352-d5de-4b2b-94d7-4a865c9512a7" containerID="a7f364b9732c96dcecb05e518b6e2f9bb214380cca68a7b780bd941f3fcae0e4" exitCode=0 Jul 08 19:49:45.419858 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:45.419394 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" event={"ID":"bfafd352-d5de-4b2b-94d7-4a865c9512a7","Type":"ContainerDied","Data":"a7f364b9732c96dcecb05e518b6e2f9bb214380cca68a7b780bd941f3fcae0e4"} Jul 08 19:49:45.419858 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:45.419452 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" event={"ID":"bfafd352-d5de-4b2b-94d7-4a865c9512a7","Type":"ContainerStarted","Data":"245107f4b89a0473c45c47a3417f3cb8eaeb8f475f66cdc4bdc3e27f417cbe55"} Jul 08 19:49:47.429726 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:47.429693 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" event={"ID":"cfc9b264-2348-476e-bc67-7368923e2087","Type":"ContainerStarted","Data":"22facf6d58b43936c54747d2670959bd981b796ac232e0d8383deeb2ca270a58"} Jul 08 19:49:47.430081 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:47.429752 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" Jul 08 19:49:47.443707 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:47.443666 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" podStartSLOduration=1.004557638 podStartE2EDuration="4.443650362s" totalImagesPullingTime="3.439092724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:49:47.442627539 +0000 UTC m=+458.607177942" watchObservedRunningTime="2026-07-08 19:49:47.443650362 +0000 UTC m=+458.608200763" Jul 08 19:49:50.442919 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:50.442882 2572 generic.go:356] "Generic (PLEG): container finished" podID="bfafd352-d5de-4b2b-94d7-4a865c9512a7" containerID="8c2d6edec1f8e78e8b802accbdd4eb362d787cb1cc7e0a5712766bb8980e3411" exitCode=0 Jul 08 19:49:50.443277 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:50.442965 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" event={"ID":"bfafd352-d5de-4b2b-94d7-4a865c9512a7","Type":"ContainerDied","Data":"8c2d6edec1f8e78e8b802accbdd4eb362d787cb1cc7e0a5712766bb8980e3411"} Jul 08 19:49:51.450129 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:51.450093 2572 generic.go:356] "Generic (PLEG): container finished" podID="bfafd352-d5de-4b2b-94d7-4a865c9512a7" containerID="b3e6cdf5e05cbaf6450f0073be1f437cd5f9f6e751d92bee0021059d0d7fca1a" exitCode=0 Jul 08 19:49:51.450555 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:51.450195 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" event={"ID":"bfafd352-d5de-4b2b-94d7-4a865c9512a7","Type":"ContainerDied","Data":"b3e6cdf5e05cbaf6450f0073be1f437cd5f9f6e751d92bee0021059d0d7fca1a"} Jul 08 19:49:52.576691 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:52.576670 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:52.687141 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:52.687115 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lnvj\" (UniqueName: \"kubernetes.io/projected/bfafd352-d5de-4b2b-94d7-4a865c9512a7-kube-api-access-9lnvj\") pod \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " Jul 08 19:49:52.687141 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:52.687145 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-bundle\") pod \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " Jul 08 19:49:52.687358 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:52.687210 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-util\") pod \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\" (UID: \"bfafd352-d5de-4b2b-94d7-4a865c9512a7\") " Jul 08 19:49:52.687553 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:52.687531 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-bundle" (OuterVolumeSpecName: "bundle") pod "bfafd352-d5de-4b2b-94d7-4a865c9512a7" (UID: "bfafd352-d5de-4b2b-94d7-4a865c9512a7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:49:52.689101 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:52.689068 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfafd352-d5de-4b2b-94d7-4a865c9512a7-kube-api-access-9lnvj" (OuterVolumeSpecName: "kube-api-access-9lnvj") pod "bfafd352-d5de-4b2b-94d7-4a865c9512a7" (UID: "bfafd352-d5de-4b2b-94d7-4a865c9512a7"). InnerVolumeSpecName "kube-api-access-9lnvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:49:52.691500 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:52.691479 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-util" (OuterVolumeSpecName: "util") pod "bfafd352-d5de-4b2b-94d7-4a865c9512a7" (UID: "bfafd352-d5de-4b2b-94d7-4a865c9512a7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:49:52.788064 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:52.788044 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lnvj\" (UniqueName: \"kubernetes.io/projected/bfafd352-d5de-4b2b-94d7-4a865c9512a7-kube-api-access-9lnvj\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:49:52.788146 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:52.788065 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-bundle\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:49:52.788146 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:52.788075 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfafd352-d5de-4b2b-94d7-4a865c9512a7-util\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:49:53.436367 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:53.436337 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xg8vv" Jul 08 19:49:53.459618 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:53.459594 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" event={"ID":"bfafd352-d5de-4b2b-94d7-4a865c9512a7","Type":"ContainerDied","Data":"245107f4b89a0473c45c47a3417f3cb8eaeb8f475f66cdc4bdc3e27f417cbe55"} Jul 08 19:49:53.459754 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:53.459623 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzff7r" Jul 08 19:49:53.459754 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:53.459630 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="245107f4b89a0473c45c47a3417f3cb8eaeb8f475f66cdc4bdc3e27f417cbe55" Jul 08 19:49:57.467943 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.467904 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs"] Jul 08 19:49:57.468420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.468309 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfafd352-d5de-4b2b-94d7-4a865c9512a7" containerName="pull" Jul 08 19:49:57.468420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.468327 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfafd352-d5de-4b2b-94d7-4a865c9512a7" containerName="pull" Jul 08 19:49:57.468420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.468340 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfafd352-d5de-4b2b-94d7-4a865c9512a7" containerName="util" Jul 08 19:49:57.468420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.468348 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfafd352-d5de-4b2b-94d7-4a865c9512a7" containerName="util" Jul 08 19:49:57.468420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.468363 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfafd352-d5de-4b2b-94d7-4a865c9512a7" containerName="extract" Jul 08 19:49:57.468420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.468372 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfafd352-d5de-4b2b-94d7-4a865c9512a7" containerName="extract" Jul 08 19:49:57.468694 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.468433 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfafd352-d5de-4b2b-94d7-4a865c9512a7" containerName="extract" Jul 08 19:49:57.492129 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.492105 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs"] Jul 08 19:49:57.492283 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.492226 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" Jul 08 19:49:57.494538 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.494514 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 19:49:57.495253 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.495229 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 19:49:57.495370 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.495287 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pfhzv\"" Jul 08 19:49:57.618043 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.618012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbl5g\" (UniqueName: \"kubernetes.io/projected/4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf-kube-api-access-pbl5g\") pod \"openshift-lws-operator-fd8ccff4c-798qs\" (UID: \"4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" Jul 08 19:49:57.618216 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.618060 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf-tmp\") pod \"openshift-lws-operator-fd8ccff4c-798qs\" (UID: \"4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" Jul 08 19:49:57.718703 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.718632 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbl5g\" (UniqueName: \"kubernetes.io/projected/4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf-kube-api-access-pbl5g\") pod \"openshift-lws-operator-fd8ccff4c-798qs\" (UID: \"4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" Jul 08 19:49:57.718837 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.718785 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf-tmp\") pod \"openshift-lws-operator-fd8ccff4c-798qs\" (UID: \"4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" Jul 08 19:49:57.719119 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.719105 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf-tmp\") pod \"openshift-lws-operator-fd8ccff4c-798qs\" (UID: \"4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" Jul 08 19:49:57.726131 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.726109 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbl5g\" (UniqueName: \"kubernetes.io/projected/4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf-kube-api-access-pbl5g\") pod \"openshift-lws-operator-fd8ccff4c-798qs\" (UID: \"4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" Jul 08 19:49:57.801274 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.801250 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" Jul 08 19:49:57.937943 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:57.937792 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs"] Jul 08 19:49:57.940424 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:49:57.940398 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d8f89a4_6bbf_4ef3_a5c7_41f875de9adf.slice/crio-f4e75b6e5fb1d00f62fcdbab28a8daeb5e31bd4b04e04edbd13a7ca84559a3aa WatchSource:0}: Error finding container f4e75b6e5fb1d00f62fcdbab28a8daeb5e31bd4b04e04edbd13a7ca84559a3aa: Status 404 returned error can't find the container with id f4e75b6e5fb1d00f62fcdbab28a8daeb5e31bd4b04e04edbd13a7ca84559a3aa Jul 08 19:49:58.482494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:49:58.482459 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" event={"ID":"4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf","Type":"ContainerStarted","Data":"f4e75b6e5fb1d00f62fcdbab28a8daeb5e31bd4b04e04edbd13a7ca84559a3aa"} Jul 08 19:50:01.497202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:01.497148 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" event={"ID":"4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf","Type":"ContainerStarted","Data":"0fef7b2f224e43fab8802d3a2dc2138b97a80c6ac9cda95a3bf29efc18ca4e90"} Jul 08 19:50:01.512663 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:01.512617 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-798qs" podStartSLOduration=1.839666105 podStartE2EDuration="4.51260404s" totalImagesPullingTime="2.672937935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:50:01.512043432 +0000 UTC m=+472.676593836" watchObservedRunningTime="2026-07-08 19:50:01.51260404 +0000 UTC m=+472.677154442" Jul 08 19:50:03.905350 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:03.905317 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm"] Jul 08 19:50:03.907729 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:03.907714 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:03.910665 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:03.910637 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 19:50:03.910778 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:03.910695 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 19:50:03.910778 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:03.910712 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-s7xwf\"" Jul 08 19:50:03.919637 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:03.919619 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm"] Jul 08 19:50:03.960532 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:03.960507 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bts\" (UniqueName: \"kubernetes.io/projected/5912900a-6b52-4769-adaf-198bf2fbd578-kube-api-access-b2bts\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:03.960655 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:03.960586 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:03.960655 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:03.960629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:04.061111 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.061082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:04.061243 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.061120 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:04.061243 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.061146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bts\" (UniqueName: \"kubernetes.io/projected/5912900a-6b52-4769-adaf-198bf2fbd578-kube-api-access-b2bts\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:04.061493 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.061474 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:04.061531 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.061490 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:04.069345 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.069325 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bts\" (UniqueName: \"kubernetes.io/projected/5912900a-6b52-4769-adaf-198bf2fbd578-kube-api-access-b2bts\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:04.215706 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.215650 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:04.336011 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.335987 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm"] Jul 08 19:50:04.338144 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:50:04.338116 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5912900a_6b52_4769_adaf_198bf2fbd578.slice/crio-8627ce16fd67cd7a81831d47234123a6e34c305fc21061443addce0701f47657 WatchSource:0}: Error finding container 8627ce16fd67cd7a81831d47234123a6e34c305fc21061443addce0701f47657: Status 404 returned error can't find the container with id 8627ce16fd67cd7a81831d47234123a6e34c305fc21061443addce0701f47657 Jul 08 19:50:04.515704 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.515638 2572 generic.go:356] "Generic (PLEG): container finished" podID="5912900a-6b52-4769-adaf-198bf2fbd578" containerID="4a968837eb301be78d3d53619de8b27ff1a3e909f6da8e60706c89fbd7821a7d" exitCode=0 Jul 08 19:50:04.515832 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.515717 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" event={"ID":"5912900a-6b52-4769-adaf-198bf2fbd578","Type":"ContainerDied","Data":"4a968837eb301be78d3d53619de8b27ff1a3e909f6da8e60706c89fbd7821a7d"} Jul 08 19:50:04.515832 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:04.515783 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" event={"ID":"5912900a-6b52-4769-adaf-198bf2fbd578","Type":"ContainerStarted","Data":"8627ce16fd67cd7a81831d47234123a6e34c305fc21061443addce0701f47657"} Jul 08 19:50:05.521385 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:05.521356 2572 generic.go:356] "Generic (PLEG): container finished" podID="5912900a-6b52-4769-adaf-198bf2fbd578" containerID="ead27292a387d25a43088a13bd988eaa20c0bcf9a2ed19f9ed367259fd37a4f0" exitCode=0 Jul 08 19:50:05.521717 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:05.521437 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" event={"ID":"5912900a-6b52-4769-adaf-198bf2fbd578","Type":"ContainerDied","Data":"ead27292a387d25a43088a13bd988eaa20c0bcf9a2ed19f9ed367259fd37a4f0"} Jul 08 19:50:06.527953 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:06.527919 2572 generic.go:356] "Generic (PLEG): container finished" podID="5912900a-6b52-4769-adaf-198bf2fbd578" containerID="756b65dff2a6fdc998f8134908c2655f38e2b52608e8340724611d11ee66ce1a" exitCode=0 Jul 08 19:50:06.528336 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:06.528004 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" event={"ID":"5912900a-6b52-4769-adaf-198bf2fbd578","Type":"ContainerDied","Data":"756b65dff2a6fdc998f8134908c2655f38e2b52608e8340724611d11ee66ce1a"} Jul 08 19:50:07.654546 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:07.654525 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:07.788889 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:07.788825 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2bts\" (UniqueName: \"kubernetes.io/projected/5912900a-6b52-4769-adaf-198bf2fbd578-kube-api-access-b2bts\") pod \"5912900a-6b52-4769-adaf-198bf2fbd578\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " Jul 08 19:50:07.788889 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:07.788857 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-util\") pod \"5912900a-6b52-4769-adaf-198bf2fbd578\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " Jul 08 19:50:07.789049 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:07.788927 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-bundle\") pod \"5912900a-6b52-4769-adaf-198bf2fbd578\" (UID: \"5912900a-6b52-4769-adaf-198bf2fbd578\") " Jul 08 19:50:07.789627 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:07.789602 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-bundle" (OuterVolumeSpecName: "bundle") pod "5912900a-6b52-4769-adaf-198bf2fbd578" (UID: "5912900a-6b52-4769-adaf-198bf2fbd578"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:50:07.790845 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:07.790821 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5912900a-6b52-4769-adaf-198bf2fbd578-kube-api-access-b2bts" (OuterVolumeSpecName: "kube-api-access-b2bts") pod "5912900a-6b52-4769-adaf-198bf2fbd578" (UID: "5912900a-6b52-4769-adaf-198bf2fbd578"). InnerVolumeSpecName "kube-api-access-b2bts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:50:07.796603 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:07.796566 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-util" (OuterVolumeSpecName: "util") pod "5912900a-6b52-4769-adaf-198bf2fbd578" (UID: "5912900a-6b52-4769-adaf-198bf2fbd578"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:50:07.890418 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:07.890389 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-bundle\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:50:07.890418 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:07.890418 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2bts\" (UniqueName: \"kubernetes.io/projected/5912900a-6b52-4769-adaf-198bf2fbd578-kube-api-access-b2bts\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:50:07.890418 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:07.890426 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5912900a-6b52-4769-adaf-198bf2fbd578-util\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:50:08.538394 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:08.538365 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" Jul 08 19:50:08.538552 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:08.538373 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afn8dsm" event={"ID":"5912900a-6b52-4769-adaf-198bf2fbd578","Type":"ContainerDied","Data":"8627ce16fd67cd7a81831d47234123a6e34c305fc21061443addce0701f47657"} Jul 08 19:50:08.538552 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:08.538480 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8627ce16fd67cd7a81831d47234123a6e34c305fc21061443addce0701f47657" Jul 08 19:50:18.076119 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.076089 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj"] Jul 08 19:50:18.076623 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.076371 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5912900a-6b52-4769-adaf-198bf2fbd578" containerName="util" Jul 08 19:50:18.076623 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.076383 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5912900a-6b52-4769-adaf-198bf2fbd578" containerName="util" Jul 08 19:50:18.076623 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.076392 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5912900a-6b52-4769-adaf-198bf2fbd578" containerName="extract" Jul 08 19:50:18.076623 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.076399 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5912900a-6b52-4769-adaf-198bf2fbd578" containerName="extract" Jul 08 19:50:18.076623 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.076410 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5912900a-6b52-4769-adaf-198bf2fbd578" containerName="pull" Jul 08 19:50:18.076623 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.076415 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5912900a-6b52-4769-adaf-198bf2fbd578" containerName="pull" Jul 08 19:50:18.076623 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.076466 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5912900a-6b52-4769-adaf-198bf2fbd578" containerName="extract" Jul 08 19:50:18.081576 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.081551 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.084426 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.084407 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 19:50:18.084426 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.084412 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 19:50:18.084601 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.084539 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vt2rp\"" Jul 08 19:50:18.084683 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.084670 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 19:50:18.084878 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.084856 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 19:50:18.094778 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.094755 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj"] Jul 08 19:50:18.162331 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.162309 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a1f5784-c030-4bb8-85ce-eb8eaa157791-manifests\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.162425 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.162336 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mc4l\" (UniqueName: \"kubernetes.io/projected/9a1f5784-c030-4bb8-85ce-eb8eaa157791-kube-api-access-9mc4l\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.162425 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.162362 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a1f5784-c030-4bb8-85ce-eb8eaa157791-tmp\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.162425 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.162383 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a1f5784-c030-4bb8-85ce-eb8eaa157791-apiservice-cert\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.162529 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.162457 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a1f5784-c030-4bb8-85ce-eb8eaa157791-webhook-cert\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.263811 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.263785 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a1f5784-c030-4bb8-85ce-eb8eaa157791-webhook-cert\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.263913 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.263835 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a1f5784-c030-4bb8-85ce-eb8eaa157791-manifests\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.263913 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.263870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mc4l\" (UniqueName: \"kubernetes.io/projected/9a1f5784-c030-4bb8-85ce-eb8eaa157791-kube-api-access-9mc4l\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.263913 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.263906 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a1f5784-c030-4bb8-85ce-eb8eaa157791-tmp\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.264064 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.263933 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a1f5784-c030-4bb8-85ce-eb8eaa157791-apiservice-cert\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.264366 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.264342 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9a1f5784-c030-4bb8-85ce-eb8eaa157791-manifests\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.266175 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.266133 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a1f5784-c030-4bb8-85ce-eb8eaa157791-tmp\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.266317 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.266295 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9a1f5784-c030-4bb8-85ce-eb8eaa157791-apiservice-cert\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.266415 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.266400 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9a1f5784-c030-4bb8-85ce-eb8eaa157791-webhook-cert\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.272534 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.272515 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mc4l\" (UniqueName: \"kubernetes.io/projected/9a1f5784-c030-4bb8-85ce-eb8eaa157791-kube-api-access-9mc4l\") pod \"opendatahub-operator-controller-manager-795d769c5f-8fssj\" (UID: \"9a1f5784-c030-4bb8-85ce-eb8eaa157791\") " pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.391373 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.391300 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:18.527958 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.527891 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj"] Jul 08 19:50:18.530177 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:50:18.530141 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1f5784_c030_4bb8_85ce_eb8eaa157791.slice/crio-ba278162b66f68e22d72f0007e70df344b9d230f09f44221b01c9f0096819ae9 WatchSource:0}: Error finding container ba278162b66f68e22d72f0007e70df344b9d230f09f44221b01c9f0096819ae9: Status 404 returned error can't find the container with id ba278162b66f68e22d72f0007e70df344b9d230f09f44221b01c9f0096819ae9 Jul 08 19:50:18.581526 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:18.581496 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" event={"ID":"9a1f5784-c030-4bb8-85ce-eb8eaa157791","Type":"ContainerStarted","Data":"ba278162b66f68e22d72f0007e70df344b9d230f09f44221b01c9f0096819ae9"} Jul 08 19:50:21.597827 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:21.597798 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" event={"ID":"9a1f5784-c030-4bb8-85ce-eb8eaa157791","Type":"ContainerStarted","Data":"e0dca518a56bd043099b90f0a37888781761c0b22ab1b9e9edc1e95d3ac43faa"} Jul 08 19:50:22.603436 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:22.603402 2572 generic.go:356] "Generic (PLEG): container finished" podID="9a1f5784-c030-4bb8-85ce-eb8eaa157791" containerID="e0dca518a56bd043099b90f0a37888781761c0b22ab1b9e9edc1e95d3ac43faa" exitCode=0 Jul 08 19:50:22.603783 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:22.603480 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" event={"ID":"9a1f5784-c030-4bb8-85ce-eb8eaa157791","Type":"ContainerDied","Data":"e0dca518a56bd043099b90f0a37888781761c0b22ab1b9e9edc1e95d3ac43faa"} Jul 08 19:50:23.610140 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:23.610102 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" event={"ID":"9a1f5784-c030-4bb8-85ce-eb8eaa157791","Type":"ContainerStarted","Data":"8f4d01f372a08fe1171a87af39204fff97b2ec313b0845f549eeaec3d236d2ce"} Jul 08 19:50:23.610580 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:23.610229 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:23.633355 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:23.633304 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" podStartSLOduration=1.303358174 podStartE2EDuration="5.633288161s" totalImagesPullingTime="3.32733679s" totalInitContainerRuntime="1.002593197s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:50:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:50:23.631931148 +0000 UTC m=+494.796481570" watchObservedRunningTime="2026-07-08 19:50:23.633288161 +0000 UTC m=+494.797838563" Jul 08 19:50:34.617608 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:34.617573 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-795d769c5f-8fssj" Jul 08 19:50:38.119238 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.119194 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd"] Jul 08 19:50:38.123839 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.123819 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.127131 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.127113 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dv96p\"" Jul 08 19:50:38.127283 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.127170 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 19:50:38.127954 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.127934 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 19:50:38.136612 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.136591 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd"] Jul 08 19:50:38.231762 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.231725 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a2e2d6-aca3-42ac-83f5-7978e902a612-tls-certs\") pod \"kube-auth-proxy-779dfd48cc-b8ktd\" (UID: \"96a2e2d6-aca3-42ac-83f5-7978e902a612\") " pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.231935 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.231782 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96a2e2d6-aca3-42ac-83f5-7978e902a612-tmp\") pod \"kube-auth-proxy-779dfd48cc-b8ktd\" (UID: \"96a2e2d6-aca3-42ac-83f5-7978e902a612\") " pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.231935 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.231809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642pq\" (UniqueName: \"kubernetes.io/projected/96a2e2d6-aca3-42ac-83f5-7978e902a612-kube-api-access-642pq\") pod \"kube-auth-proxy-779dfd48cc-b8ktd\" (UID: \"96a2e2d6-aca3-42ac-83f5-7978e902a612\") " pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.333078 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.333042 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96a2e2d6-aca3-42ac-83f5-7978e902a612-tmp\") pod \"kube-auth-proxy-779dfd48cc-b8ktd\" (UID: \"96a2e2d6-aca3-42ac-83f5-7978e902a612\") " pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.333280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.333103 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642pq\" (UniqueName: \"kubernetes.io/projected/96a2e2d6-aca3-42ac-83f5-7978e902a612-kube-api-access-642pq\") pod \"kube-auth-proxy-779dfd48cc-b8ktd\" (UID: \"96a2e2d6-aca3-42ac-83f5-7978e902a612\") " pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.333280 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.333186 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a2e2d6-aca3-42ac-83f5-7978e902a612-tls-certs\") pod \"kube-auth-proxy-779dfd48cc-b8ktd\" (UID: \"96a2e2d6-aca3-42ac-83f5-7978e902a612\") " pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.335609 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.335574 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96a2e2d6-aca3-42ac-83f5-7978e902a612-tmp\") pod \"kube-auth-proxy-779dfd48cc-b8ktd\" (UID: \"96a2e2d6-aca3-42ac-83f5-7978e902a612\") " pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.335724 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.335700 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a2e2d6-aca3-42ac-83f5-7978e902a612-tls-certs\") pod \"kube-auth-proxy-779dfd48cc-b8ktd\" (UID: \"96a2e2d6-aca3-42ac-83f5-7978e902a612\") " pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.345267 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.345244 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-642pq\" (UniqueName: \"kubernetes.io/projected/96a2e2d6-aca3-42ac-83f5-7978e902a612-kube-api-access-642pq\") pod \"kube-auth-proxy-779dfd48cc-b8ktd\" (UID: \"96a2e2d6-aca3-42ac-83f5-7978e902a612\") " pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.434096 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.434008 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" Jul 08 19:50:38.578742 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.578711 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd"] Jul 08 19:50:38.582219 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:50:38.582186 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a2e2d6_aca3_42ac_83f5_7978e902a612.slice/crio-80af93a15697ac4904f953cfafb2ed44818f943067aca1ba3e02b17368889f0e WatchSource:0}: Error finding container 80af93a15697ac4904f953cfafb2ed44818f943067aca1ba3e02b17368889f0e: Status 404 returned error can't find the container with id 80af93a15697ac4904f953cfafb2ed44818f943067aca1ba3e02b17368889f0e Jul 08 19:50:38.684848 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:38.684753 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" event={"ID":"96a2e2d6-aca3-42ac-83f5-7978e902a612","Type":"ContainerStarted","Data":"80af93a15697ac4904f953cfafb2ed44818f943067aca1ba3e02b17368889f0e"} Jul 08 19:50:39.192395 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.192354 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-c9zjq"] Jul 08 19:50:39.197471 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.197446 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" Jul 08 19:50:39.200186 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.200144 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 08 19:50:39.207639 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.207480 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-c9zjq"] Jul 08 19:50:39.241537 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.241499 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxb5m\" (UniqueName: \"kubernetes.io/projected/85de8a0b-8359-42c6-aa06-4695f0bf4733-kube-api-access-lxb5m\") pod \"model-serving-api-54d89bd5fc-c9zjq\" (UID: \"85de8a0b-8359-42c6-aa06-4695f0bf4733\") " pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" Jul 08 19:50:39.241684 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.241562 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de8a0b-8359-42c6-aa06-4695f0bf4733-tls-certs\") pod \"model-serving-api-54d89bd5fc-c9zjq\" (UID: \"85de8a0b-8359-42c6-aa06-4695f0bf4733\") " pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" Jul 08 19:50:39.342717 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.342686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxb5m\" (UniqueName: \"kubernetes.io/projected/85de8a0b-8359-42c6-aa06-4695f0bf4733-kube-api-access-lxb5m\") pod \"model-serving-api-54d89bd5fc-c9zjq\" (UID: \"85de8a0b-8359-42c6-aa06-4695f0bf4733\") " pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" Jul 08 19:50:39.342886 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.342761 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de8a0b-8359-42c6-aa06-4695f0bf4733-tls-certs\") pod \"model-serving-api-54d89bd5fc-c9zjq\" (UID: \"85de8a0b-8359-42c6-aa06-4695f0bf4733\") " pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" Jul 08 19:50:39.345830 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.345772 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85de8a0b-8359-42c6-aa06-4695f0bf4733-tls-certs\") pod \"model-serving-api-54d89bd5fc-c9zjq\" (UID: \"85de8a0b-8359-42c6-aa06-4695f0bf4733\") " pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" Jul 08 19:50:39.352514 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.352450 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxb5m\" (UniqueName: \"kubernetes.io/projected/85de8a0b-8359-42c6-aa06-4695f0bf4733-kube-api-access-lxb5m\") pod \"model-serving-api-54d89bd5fc-c9zjq\" (UID: \"85de8a0b-8359-42c6-aa06-4695f0bf4733\") " pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" Jul 08 19:50:39.510469 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.510385 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" Jul 08 19:50:39.676684 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.676659 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-c9zjq"] Jul 08 19:50:39.678933 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:50:39.678900 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85de8a0b_8359_42c6_aa06_4695f0bf4733.slice/crio-4626d70d535ed7df0233b0828609d1582dbdc90677ac0503f80be01dca14ad59 WatchSource:0}: Error finding container 4626d70d535ed7df0233b0828609d1582dbdc90677ac0503f80be01dca14ad59: Status 404 returned error can't find the container with id 4626d70d535ed7df0233b0828609d1582dbdc90677ac0503f80be01dca14ad59 Jul 08 19:50:39.691471 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:39.691432 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" event={"ID":"85de8a0b-8359-42c6-aa06-4695f0bf4733","Type":"ContainerStarted","Data":"4626d70d535ed7df0233b0828609d1582dbdc90677ac0503f80be01dca14ad59"} Jul 08 19:50:42.710408 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:42.710370 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" event={"ID":"85de8a0b-8359-42c6-aa06-4695f0bf4733","Type":"ContainerStarted","Data":"6b79c2cdc90bb52726cf7432018e60d04361e0b6c8c243305a8b5e036605631a"} Jul 08 19:50:42.710871 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:42.710438 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" Jul 08 19:50:42.712014 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:42.711992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" event={"ID":"96a2e2d6-aca3-42ac-83f5-7978e902a612","Type":"ContainerStarted","Data":"b4ccc3f59cd4334cfb37cb4a3506cc106eddf43d394055ffa0dc3b8430d9d4f8"} Jul 08 19:50:42.728435 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:42.728386 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" podStartSLOduration=1.316113822 podStartE2EDuration="3.72836873s" totalImagesPullingTime="2.412254908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:50:42.726921166 +0000 UTC m=+513.891471569" watchObservedRunningTime="2026-07-08 19:50:42.72836873 +0000 UTC m=+513.892919135" Jul 08 19:50:42.743111 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:42.743063 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-779dfd48cc-b8ktd" podStartSLOduration=1.233633867 podStartE2EDuration="4.743045685s" totalImagesPullingTime="3.509411818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:50:42.742307089 +0000 UTC m=+513.906857493" watchObservedRunningTime="2026-07-08 19:50:42.743045685 +0000 UTC m=+513.907596089" Jul 08 19:50:45.511656 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.511617 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p"] Jul 08 19:50:45.515398 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.515374 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.518444 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.518424 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 19:50:45.518772 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.518750 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 19:50:45.518837 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.518771 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 19:50:45.518837 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.518750 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7dkwx\"" Jul 08 19:50:45.525212 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.525190 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p"] Jul 08 19:50:45.602584 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.602552 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f23d8219-e7b2-4283-ad02-786e694e1aad-manager-config\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.602727 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.602607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlrp\" (UniqueName: \"kubernetes.io/projected/f23d8219-e7b2-4283-ad02-786e694e1aad-kube-api-access-9xlrp\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.602727 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.602647 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f23d8219-e7b2-4283-ad02-786e694e1aad-cert\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.602727 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.602675 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f23d8219-e7b2-4283-ad02-786e694e1aad-metrics-cert\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.703583 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.703552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f23d8219-e7b2-4283-ad02-786e694e1aad-manager-config\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.703759 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.703615 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlrp\" (UniqueName: \"kubernetes.io/projected/f23d8219-e7b2-4283-ad02-786e694e1aad-kube-api-access-9xlrp\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.703759 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.703654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f23d8219-e7b2-4283-ad02-786e694e1aad-cert\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.703759 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.703690 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f23d8219-e7b2-4283-ad02-786e694e1aad-metrics-cert\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.704256 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.704236 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f23d8219-e7b2-4283-ad02-786e694e1aad-manager-config\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.706138 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.706115 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f23d8219-e7b2-4283-ad02-786e694e1aad-cert\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.706230 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.706174 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f23d8219-e7b2-4283-ad02-786e694e1aad-metrics-cert\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.715318 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.715291 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlrp\" (UniqueName: \"kubernetes.io/projected/f23d8219-e7b2-4283-ad02-786e694e1aad-kube-api-access-9xlrp\") pod \"lws-controller-manager-7cffbd996-hql4p\" (UID: \"f23d8219-e7b2-4283-ad02-786e694e1aad\") " pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:45.824732 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:45.824701 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:46.005535 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:46.005510 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p"] Jul 08 19:50:46.007363 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:50:46.007339 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23d8219_e7b2_4283_ad02_786e694e1aad.slice/crio-c9cbd886f8ecf6da6237d8116d5c36b2fb389e06dbdc06d20386c0360d2076e5 WatchSource:0}: Error finding container c9cbd886f8ecf6da6237d8116d5c36b2fb389e06dbdc06d20386c0360d2076e5: Status 404 returned error can't find the container with id c9cbd886f8ecf6da6237d8116d5c36b2fb389e06dbdc06d20386c0360d2076e5 Jul 08 19:50:46.732549 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:46.732512 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" event={"ID":"f23d8219-e7b2-4283-ad02-786e694e1aad","Type":"ContainerStarted","Data":"c9cbd886f8ecf6da6237d8116d5c36b2fb389e06dbdc06d20386c0360d2076e5"} Jul 08 19:50:47.738653 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:47.738568 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" event={"ID":"f23d8219-e7b2-4283-ad02-786e694e1aad","Type":"ContainerStarted","Data":"bff28ded6bd2e92287ecd3f3fc1fe3876845825399dbe49e9f99a97322c83dc6"} Jul 08 19:50:47.739005 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:47.738688 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:50:47.764645 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:47.764597 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" podStartSLOduration=1.427134328 podStartE2EDuration="2.764580307s" totalImagesPullingTime="1.337445979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:50:47.762064633 +0000 UTC m=+518.926615036" watchObservedRunningTime="2026-07-08 19:50:47.764580307 +0000 UTC m=+518.929130713" Jul 08 19:50:53.720272 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:53.720246 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-c9zjq" Jul 08 19:50:58.747119 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:50:58.747088 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cffbd996-hql4p" Jul 08 19:51:54.414849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.414818 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94"] Jul 08 19:51:54.417335 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.417319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.419485 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.419466 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xl2pr\"" Jul 08 19:51:54.419571 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.419526 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 19:51:54.419571 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.419544 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 19:51:54.425650 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.425629 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94"] Jul 08 19:51:54.546687 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.546656 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.546812 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.546715 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.546812 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.546736 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7pc\" (UniqueName: \"kubernetes.io/projected/bc3a6113-f959-4f56-949b-8bac13e57cce-kube-api-access-gw7pc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.647645 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.647622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.647764 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.647650 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7pc\" (UniqueName: \"kubernetes.io/projected/bc3a6113-f959-4f56-949b-8bac13e57cce-kube-api-access-gw7pc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.647764 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.647688 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.648035 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.648018 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.648101 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.648037 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.655647 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.655620 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7pc\" (UniqueName: \"kubernetes.io/projected/bc3a6113-f959-4f56-949b-8bac13e57cce-kube-api-access-gw7pc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.726742 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.726694 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:54.853453 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:54.853433 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94"] Jul 08 19:51:54.855258 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:51:54.855228 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3a6113_f959_4f56_949b_8bac13e57cce.slice/crio-b49e1de9d293cfe3d88b9b15b4b783902ea14aabb0a98320619c1592bca0686c WatchSource:0}: Error finding container b49e1de9d293cfe3d88b9b15b4b783902ea14aabb0a98320619c1592bca0686c: Status 404 returned error can't find the container with id b49e1de9d293cfe3d88b9b15b4b783902ea14aabb0a98320619c1592bca0686c Jul 08 19:51:55.014874 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.014819 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk"] Jul 08 19:51:55.017512 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.017495 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.025970 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.025947 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk"] Jul 08 19:51:55.045311 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.045285 2572 generic.go:356] "Generic (PLEG): container finished" podID="bc3a6113-f959-4f56-949b-8bac13e57cce" containerID="3593373962ef534453898623b35d278941c46356bcf2e7284322d0078f0b66e2" exitCode=0 Jul 08 19:51:55.045394 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.045354 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" event={"ID":"bc3a6113-f959-4f56-949b-8bac13e57cce","Type":"ContainerDied","Data":"3593373962ef534453898623b35d278941c46356bcf2e7284322d0078f0b66e2"} Jul 08 19:51:55.045394 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.045381 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" event={"ID":"bc3a6113-f959-4f56-949b-8bac13e57cce","Type":"ContainerStarted","Data":"b49e1de9d293cfe3d88b9b15b4b783902ea14aabb0a98320619c1592bca0686c"} Jul 08 19:51:55.152645 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.152622 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.152765 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.152672 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tc9\" (UniqueName: \"kubernetes.io/projected/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-kube-api-access-q9tc9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.152765 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.152754 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.254046 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.254018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tc9\" (UniqueName: \"kubernetes.io/projected/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-kube-api-access-q9tc9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.254185 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.254067 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.254185 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.254101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.254470 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.254450 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.254506 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.254466 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.262077 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.262051 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tc9\" (UniqueName: \"kubernetes.io/projected/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-kube-api-access-q9tc9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.326567 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.326548 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:55.413668 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.413641 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z"] Jul 08 19:51:55.416242 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.416220 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.426575 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.426551 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z"] Jul 08 19:51:55.454480 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.454457 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk"] Jul 08 19:51:55.456545 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:51:55.456519 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a66ea7e_0861_43ba_bdb6_f5bfd61573cf.slice/crio-6fcc0131a78711f9bb991febb8f2995e2fbb402f685703996bb4cebe87e36f2f WatchSource:0}: Error finding container 6fcc0131a78711f9bb991febb8f2995e2fbb402f685703996bb4cebe87e36f2f: Status 404 returned error can't find the container with id 6fcc0131a78711f9bb991febb8f2995e2fbb402f685703996bb4cebe87e36f2f Jul 08 19:51:55.556955 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.556928 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.557052 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.556965 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsg7g\" (UniqueName: \"kubernetes.io/projected/9a854af9-aeb2-4f8f-a200-edfb4a972e66-kube-api-access-bsg7g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.557052 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.557047 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.657908 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.657832 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.657908 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.657874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsg7g\" (UniqueName: \"kubernetes.io/projected/9a854af9-aeb2-4f8f-a200-edfb4a972e66-kube-api-access-bsg7g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.658139 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.657952 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.658341 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.658319 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.658381 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.658328 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.666299 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.666277 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsg7g\" (UniqueName: \"kubernetes.io/projected/9a854af9-aeb2-4f8f-a200-edfb4a972e66-kube-api-access-bsg7g\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.729484 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.729462 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:51:55.811526 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.811495 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf"] Jul 08 19:51:55.815214 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.815178 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:55.824517 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.823649 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf"] Jul 08 19:51:55.869213 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.869193 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z"] Jul 08 19:51:55.870744 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:51:55.870721 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a854af9_aeb2_4f8f_a200_edfb4a972e66.slice/crio-c5036fc25930f1d836d145286308c9dcf41573e708fee7439513101ea6bc73f8 WatchSource:0}: Error finding container c5036fc25930f1d836d145286308c9dcf41573e708fee7439513101ea6bc73f8: Status 404 returned error can't find the container with id c5036fc25930f1d836d145286308c9dcf41573e708fee7439513101ea6bc73f8 Jul 08 19:51:55.960481 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.960452 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:55.960590 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.960507 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:55.960590 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:55.960574 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fvtd\" (UniqueName: \"kubernetes.io/projected/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-kube-api-access-7fvtd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:56.052689 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.052659 2572 generic.go:356] "Generic (PLEG): container finished" podID="9a854af9-aeb2-4f8f-a200-edfb4a972e66" containerID="6834496e05a2dc6231c3fddfee19782065e666f28a1655a102731156701829af" exitCode=0 Jul 08 19:51:56.052822 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.052755 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" event={"ID":"9a854af9-aeb2-4f8f-a200-edfb4a972e66","Type":"ContainerDied","Data":"6834496e05a2dc6231c3fddfee19782065e666f28a1655a102731156701829af"} Jul 08 19:51:56.052822 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.052815 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" event={"ID":"9a854af9-aeb2-4f8f-a200-edfb4a972e66","Type":"ContainerStarted","Data":"c5036fc25930f1d836d145286308c9dcf41573e708fee7439513101ea6bc73f8"} Jul 08 19:51:56.054854 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.054832 2572 generic.go:356] "Generic (PLEG): container finished" podID="8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" containerID="3b3d5b559e762ceab6e37e8ab3d22b7e5a436accbd9ba8f7a6b387a55944c5d5" exitCode=0 Jul 08 19:51:56.054971 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.054942 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" event={"ID":"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf","Type":"ContainerDied","Data":"3b3d5b559e762ceab6e37e8ab3d22b7e5a436accbd9ba8f7a6b387a55944c5d5"} Jul 08 19:51:56.055036 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.054981 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" event={"ID":"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf","Type":"ContainerStarted","Data":"6fcc0131a78711f9bb991febb8f2995e2fbb402f685703996bb4cebe87e36f2f"} Jul 08 19:51:56.057579 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.057531 2572 generic.go:356] "Generic (PLEG): container finished" podID="bc3a6113-f959-4f56-949b-8bac13e57cce" containerID="1e87f01c78b987e28adc6c82394603d7e7cadc30b695e5a85ee765a49201bc78" exitCode=0 Jul 08 19:51:56.057579 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.057577 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" event={"ID":"bc3a6113-f959-4f56-949b-8bac13e57cce","Type":"ContainerDied","Data":"1e87f01c78b987e28adc6c82394603d7e7cadc30b695e5a85ee765a49201bc78"} Jul 08 19:51:56.061310 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.061284 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:56.061310 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.061306 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:56.061418 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.061329 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fvtd\" (UniqueName: \"kubernetes.io/projected/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-kube-api-access-7fvtd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:56.061707 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.061687 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:56.061767 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.061694 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:56.069475 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.069458 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fvtd\" (UniqueName: \"kubernetes.io/projected/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-kube-api-access-7fvtd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:56.129261 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.129231 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:51:56.255579 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:56.255555 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf"] Jul 08 19:51:56.257870 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:51:56.257847 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1bd3fd_1a07_47e2_8bea_847a456bdaf2.slice/crio-583c6ca375b59c102744b3c9324074fdb037b09403aa60442cf62eaf92e8e680 WatchSource:0}: Error finding container 583c6ca375b59c102744b3c9324074fdb037b09403aa60442cf62eaf92e8e680: Status 404 returned error can't find the container with id 583c6ca375b59c102744b3c9324074fdb037b09403aa60442cf62eaf92e8e680 Jul 08 19:51:57.064012 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:57.063933 2572 generic.go:356] "Generic (PLEG): container finished" podID="8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" containerID="c8fc3b302b234a491c4bdea13a44983f7abe8af4ff8292bed2c7a5d2d8d23db6" exitCode=0 Jul 08 19:51:57.064469 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:57.064014 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" event={"ID":"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf","Type":"ContainerDied","Data":"c8fc3b302b234a491c4bdea13a44983f7abe8af4ff8292bed2c7a5d2d8d23db6"} Jul 08 19:51:57.066235 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:57.066212 2572 generic.go:356] "Generic (PLEG): container finished" podID="bc3a6113-f959-4f56-949b-8bac13e57cce" containerID="052f51cd92bd43efaad55c91031ee35b4fe862a956a550d5779b6fbc3b8d6c8a" exitCode=0 Jul 08 19:51:57.066353 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:57.066321 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" event={"ID":"bc3a6113-f959-4f56-949b-8bac13e57cce","Type":"ContainerDied","Data":"052f51cd92bd43efaad55c91031ee35b4fe862a956a550d5779b6fbc3b8d6c8a"} Jul 08 19:51:57.067934 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:57.067911 2572 generic.go:356] "Generic (PLEG): container finished" podID="bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" containerID="e56032af26e9969612fb9de71ee8c9e00fd74a8675b9000468139cb883f71c84" exitCode=0 Jul 08 19:51:57.068023 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:57.067962 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" event={"ID":"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2","Type":"ContainerDied","Data":"e56032af26e9969612fb9de71ee8c9e00fd74a8675b9000468139cb883f71c84"} Jul 08 19:51:57.068023 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:57.068004 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" event={"ID":"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2","Type":"ContainerStarted","Data":"583c6ca375b59c102744b3c9324074fdb037b09403aa60442cf62eaf92e8e680"} Jul 08 19:51:58.079397 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.079362 2572 generic.go:356] "Generic (PLEG): container finished" podID="8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" containerID="93a27d018dd282e63c6a33b66dc8288fc90d346b4d30fa631d64161d5cc899e4" exitCode=0 Jul 08 19:51:58.079700 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.079432 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" event={"ID":"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf","Type":"ContainerDied","Data":"93a27d018dd282e63c6a33b66dc8288fc90d346b4d30fa631d64161d5cc899e4"} Jul 08 19:51:58.081185 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.081152 2572 generic.go:356] "Generic (PLEG): container finished" podID="bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" containerID="ce0dd823217528d633ec1cfd8c61a3156265776bc17ba0ccd036282f26344072" exitCode=0 Jul 08 19:51:58.081262 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.081236 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" event={"ID":"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2","Type":"ContainerDied","Data":"ce0dd823217528d633ec1cfd8c61a3156265776bc17ba0ccd036282f26344072"} Jul 08 19:51:58.083017 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.082994 2572 generic.go:356] "Generic (PLEG): container finished" podID="9a854af9-aeb2-4f8f-a200-edfb4a972e66" containerID="e9a17912471a9c9a670c4f104ecdbeb43c2780a271cea358bf0a820c64ae0ee2" exitCode=0 Jul 08 19:51:58.083135 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.083074 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" event={"ID":"9a854af9-aeb2-4f8f-a200-edfb4a972e66","Type":"ContainerDied","Data":"e9a17912471a9c9a670c4f104ecdbeb43c2780a271cea358bf0a820c64ae0ee2"} Jul 08 19:51:58.209717 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.209625 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:58.381497 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.381462 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-bundle\") pod \"bc3a6113-f959-4f56-949b-8bac13e57cce\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " Jul 08 19:51:58.381497 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.381500 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-util\") pod \"bc3a6113-f959-4f56-949b-8bac13e57cce\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " Jul 08 19:51:58.381690 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.381564 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw7pc\" (UniqueName: \"kubernetes.io/projected/bc3a6113-f959-4f56-949b-8bac13e57cce-kube-api-access-gw7pc\") pod \"bc3a6113-f959-4f56-949b-8bac13e57cce\" (UID: \"bc3a6113-f959-4f56-949b-8bac13e57cce\") " Jul 08 19:51:58.381965 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.381934 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-bundle" (OuterVolumeSpecName: "bundle") pod "bc3a6113-f959-4f56-949b-8bac13e57cce" (UID: "bc3a6113-f959-4f56-949b-8bac13e57cce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:51:58.383888 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.383855 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3a6113-f959-4f56-949b-8bac13e57cce-kube-api-access-gw7pc" (OuterVolumeSpecName: "kube-api-access-gw7pc") pod "bc3a6113-f959-4f56-949b-8bac13e57cce" (UID: "bc3a6113-f959-4f56-949b-8bac13e57cce"). InnerVolumeSpecName "kube-api-access-gw7pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:51:58.387105 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.387084 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-util" (OuterVolumeSpecName: "util") pod "bc3a6113-f959-4f56-949b-8bac13e57cce" (UID: "bc3a6113-f959-4f56-949b-8bac13e57cce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:51:58.482526 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.482470 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7pc\" (UniqueName: \"kubernetes.io/projected/bc3a6113-f959-4f56-949b-8bac13e57cce-kube-api-access-gw7pc\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:51:58.482526 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.482493 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-bundle\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:51:58.482526 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:58.482503 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc3a6113-f959-4f56-949b-8bac13e57cce-util\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:51:59.089667 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.089641 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" Jul 08 19:51:59.090076 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.089674 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94" event={"ID":"bc3a6113-f959-4f56-949b-8bac13e57cce","Type":"ContainerDied","Data":"b49e1de9d293cfe3d88b9b15b4b783902ea14aabb0a98320619c1592bca0686c"} Jul 08 19:51:59.090076 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.089723 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b49e1de9d293cfe3d88b9b15b4b783902ea14aabb0a98320619c1592bca0686c" Jul 08 19:51:59.091732 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.091698 2572 generic.go:356] "Generic (PLEG): container finished" podID="bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" containerID="db4a35d06a95777e451adc3ab26f74415e90cad979bffa3c361962d9cc9309f7" exitCode=0 Jul 08 19:51:59.091866 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.091772 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" event={"ID":"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2","Type":"ContainerDied","Data":"db4a35d06a95777e451adc3ab26f74415e90cad979bffa3c361962d9cc9309f7"} Jul 08 19:51:59.093949 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.093922 2572 generic.go:356] "Generic (PLEG): container finished" podID="9a854af9-aeb2-4f8f-a200-edfb4a972e66" containerID="60938adc5fbf59c396c2f43020e07c1ea097dd8854fb432ee37c35a9df5a0335" exitCode=0 Jul 08 19:51:59.094064 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.093997 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" event={"ID":"9a854af9-aeb2-4f8f-a200-edfb4a972e66","Type":"ContainerDied","Data":"60938adc5fbf59c396c2f43020e07c1ea097dd8854fb432ee37c35a9df5a0335"} Jul 08 19:51:59.219563 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.219541 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:51:59.287927 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.287899 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9tc9\" (UniqueName: \"kubernetes.io/projected/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-kube-api-access-q9tc9\") pod \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " Jul 08 19:51:59.288059 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.287943 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-util\") pod \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " Jul 08 19:51:59.288059 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.287967 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-bundle\") pod \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\" (UID: \"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf\") " Jul 08 19:51:59.288430 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.288406 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-bundle" (OuterVolumeSpecName: "bundle") pod "8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" (UID: "8a66ea7e-0861-43ba-bdb6-f5bfd61573cf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:51:59.290032 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.290004 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-kube-api-access-q9tc9" (OuterVolumeSpecName: "kube-api-access-q9tc9") pod "8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" (UID: "8a66ea7e-0861-43ba-bdb6-f5bfd61573cf"). InnerVolumeSpecName "kube-api-access-q9tc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:51:59.293447 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.293415 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-util" (OuterVolumeSpecName: "util") pod "8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" (UID: "8a66ea7e-0861-43ba-bdb6-f5bfd61573cf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:51:59.388618 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.388560 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9tc9\" (UniqueName: \"kubernetes.io/projected/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-kube-api-access-q9tc9\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:51:59.388618 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.388587 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-util\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:51:59.388618 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:51:59.388597 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a66ea7e-0861-43ba-bdb6-f5bfd61573cf-bundle\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:00.100312 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.100286 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" Jul 08 19:52:00.100677 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.100286 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk" event={"ID":"8a66ea7e-0861-43ba-bdb6-f5bfd61573cf","Type":"ContainerDied","Data":"6fcc0131a78711f9bb991febb8f2995e2fbb402f685703996bb4cebe87e36f2f"} Jul 08 19:52:00.100677 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.100410 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fcc0131a78711f9bb991febb8f2995e2fbb402f685703996bb4cebe87e36f2f" Jul 08 19:52:00.225004 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.224982 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:52:00.256171 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.256143 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:52:00.294943 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.294908 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fvtd\" (UniqueName: \"kubernetes.io/projected/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-kube-api-access-7fvtd\") pod \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " Jul 08 19:52:00.294943 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.294946 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-util\") pod \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " Jul 08 19:52:00.295102 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.294991 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-bundle\") pod \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " Jul 08 19:52:00.295102 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.295010 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsg7g\" (UniqueName: \"kubernetes.io/projected/9a854af9-aeb2-4f8f-a200-edfb4a972e66-kube-api-access-bsg7g\") pod \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " Jul 08 19:52:00.295102 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.295069 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-util\") pod \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\" (UID: \"9a854af9-aeb2-4f8f-a200-edfb4a972e66\") " Jul 08 19:52:00.295255 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.295119 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-bundle\") pod \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\" (UID: \"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2\") " Jul 08 19:52:00.295787 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.295665 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-bundle" (OuterVolumeSpecName: "bundle") pod "9a854af9-aeb2-4f8f-a200-edfb4a972e66" (UID: "9a854af9-aeb2-4f8f-a200-edfb4a972e66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:52:00.295946 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.295899 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-bundle" (OuterVolumeSpecName: "bundle") pod "bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" (UID: "bb1bd3fd-1a07-47e2-8bea-847a456bdaf2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:52:00.297104 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.297077 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-kube-api-access-7fvtd" (OuterVolumeSpecName: "kube-api-access-7fvtd") pod "bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" (UID: "bb1bd3fd-1a07-47e2-8bea-847a456bdaf2"). InnerVolumeSpecName "kube-api-access-7fvtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:52:00.297237 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.297186 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a854af9-aeb2-4f8f-a200-edfb4a972e66-kube-api-access-bsg7g" (OuterVolumeSpecName: "kube-api-access-bsg7g") pod "9a854af9-aeb2-4f8f-a200-edfb4a972e66" (UID: "9a854af9-aeb2-4f8f-a200-edfb4a972e66"). InnerVolumeSpecName "kube-api-access-bsg7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:52:00.300501 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.300471 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-util" (OuterVolumeSpecName: "util") pod "bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" (UID: "bb1bd3fd-1a07-47e2-8bea-847a456bdaf2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:52:00.302495 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.302467 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-util" (OuterVolumeSpecName: "util") pod "9a854af9-aeb2-4f8f-a200-edfb4a972e66" (UID: "9a854af9-aeb2-4f8f-a200-edfb4a972e66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:52:00.396254 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.396206 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-util\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:00.396254 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.396224 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-bundle\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:00.396254 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.396233 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fvtd\" (UniqueName: \"kubernetes.io/projected/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-kube-api-access-7fvtd\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:00.396254 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.396242 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb1bd3fd-1a07-47e2-8bea-847a456bdaf2-util\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:00.396254 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.396250 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a854af9-aeb2-4f8f-a200-edfb4a972e66-bundle\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:00.396448 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:00.396259 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsg7g\" (UniqueName: \"kubernetes.io/projected/9a854af9-aeb2-4f8f-a200-edfb4a972e66-kube-api-access-bsg7g\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:01.106399 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:01.106369 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" Jul 08 19:52:01.106399 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:01.106393 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z" event={"ID":"9a854af9-aeb2-4f8f-a200-edfb4a972e66","Type":"ContainerDied","Data":"c5036fc25930f1d836d145286308c9dcf41573e708fee7439513101ea6bc73f8"} Jul 08 19:52:01.106811 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:01.106462 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5036fc25930f1d836d145286308c9dcf41573e708fee7439513101ea6bc73f8" Jul 08 19:52:01.108329 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:01.108307 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" event={"ID":"bb1bd3fd-1a07-47e2-8bea-847a456bdaf2","Type":"ContainerDied","Data":"583c6ca375b59c102744b3c9324074fdb037b09403aa60442cf62eaf92e8e680"} Jul 08 19:52:01.108404 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:01.108348 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="583c6ca375b59c102744b3c9324074fdb037b09403aa60442cf62eaf92e8e680" Jul 08 19:52:01.108404 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:01.108352 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf" Jul 08 19:52:09.477918 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:09.477887 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 19:52:09.485194 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:09.485143 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 19:52:23.439489 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:23.439458 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9bd8bbd4-lzmgh"] Jul 08 19:52:48.467506 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.467470 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7c9bd8bbd4-lzmgh" podUID="b838c65f-309a-4939-92c2-b0a10b0b1273" containerName="console" containerID="cri-o://c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1" gracePeriod=15 Jul 08 19:52:48.703382 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.703360 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:52:48.769097 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769074 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-serving-cert\") pod \"b838c65f-309a-4939-92c2-b0a10b0b1273\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " Jul 08 19:52:48.769238 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769136 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-oauth-config\") pod \"b838c65f-309a-4939-92c2-b0a10b0b1273\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " Jul 08 19:52:48.769238 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769198 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-console-config\") pod \"b838c65f-309a-4939-92c2-b0a10b0b1273\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " Jul 08 19:52:48.769354 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769248 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-trusted-ca-bundle\") pod \"b838c65f-309a-4939-92c2-b0a10b0b1273\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " Jul 08 19:52:48.769354 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769284 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-service-ca\") pod \"b838c65f-309a-4939-92c2-b0a10b0b1273\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " Jul 08 19:52:48.769354 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769339 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mwbl\" (UniqueName: \"kubernetes.io/projected/b838c65f-309a-4939-92c2-b0a10b0b1273-kube-api-access-5mwbl\") pod \"b838c65f-309a-4939-92c2-b0a10b0b1273\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " Jul 08 19:52:48.769499 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769376 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-oauth-serving-cert\") pod \"b838c65f-309a-4939-92c2-b0a10b0b1273\" (UID: \"b838c65f-309a-4939-92c2-b0a10b0b1273\") " Jul 08 19:52:48.769704 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769676 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-console-config" (OuterVolumeSpecName: "console-config") pod "b838c65f-309a-4939-92c2-b0a10b0b1273" (UID: "b838c65f-309a-4939-92c2-b0a10b0b1273"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:52:48.769798 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769702 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b838c65f-309a-4939-92c2-b0a10b0b1273" (UID: "b838c65f-309a-4939-92c2-b0a10b0b1273"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:52:48.769856 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769820 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-service-ca" (OuterVolumeSpecName: "service-ca") pod "b838c65f-309a-4939-92c2-b0a10b0b1273" (UID: "b838c65f-309a-4939-92c2-b0a10b0b1273"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:52:48.769856 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.769823 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b838c65f-309a-4939-92c2-b0a10b0b1273" (UID: "b838c65f-309a-4939-92c2-b0a10b0b1273"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:52:48.771266 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.771237 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b838c65f-309a-4939-92c2-b0a10b0b1273" (UID: "b838c65f-309a-4939-92c2-b0a10b0b1273"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:52:48.771368 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.771285 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b838c65f-309a-4939-92c2-b0a10b0b1273" (UID: "b838c65f-309a-4939-92c2-b0a10b0b1273"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:52:48.771639 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.771624 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b838c65f-309a-4939-92c2-b0a10b0b1273-kube-api-access-5mwbl" (OuterVolumeSpecName: "kube-api-access-5mwbl") pod "b838c65f-309a-4939-92c2-b0a10b0b1273" (UID: "b838c65f-309a-4939-92c2-b0a10b0b1273"). InnerVolumeSpecName "kube-api-access-5mwbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:52:48.870742 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.870714 2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-serving-cert\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:48.870742 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.870736 2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b838c65f-309a-4939-92c2-b0a10b0b1273-console-oauth-config\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:48.870742 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.870745 2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-console-config\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:48.870934 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.870754 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-trusted-ca-bundle\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:48.870934 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.870764 2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:48.870934 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.870774 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mwbl\" (UniqueName: \"kubernetes.io/projected/b838c65f-309a-4939-92c2-b0a10b0b1273-kube-api-access-5mwbl\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:48.870934 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:48.870782 2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b838c65f-309a-4939-92c2-b0a10b0b1273-oauth-serving-cert\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:52:49.332240 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:49.332204 2572 generic.go:356] "Generic (PLEG): container finished" podID="b838c65f-309a-4939-92c2-b0a10b0b1273" containerID="c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1" exitCode=0 Jul 08 19:52:49.332420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:49.332264 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9bd8bbd4-lzmgh" Jul 08 19:52:49.332420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:49.332285 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd8bbd4-lzmgh" event={"ID":"b838c65f-309a-4939-92c2-b0a10b0b1273","Type":"ContainerDied","Data":"c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1"} Jul 08 19:52:49.332420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:49.332383 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9bd8bbd4-lzmgh" event={"ID":"b838c65f-309a-4939-92c2-b0a10b0b1273","Type":"ContainerDied","Data":"861d88c0d7e050f23c4c41ab9c8625cd3f8a9efa184625210eda1273a927d4ea"} Jul 08 19:52:49.332420 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:49.332410 2572 scope.go:117] "RemoveContainer" containerID="c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1" Jul 08 19:52:49.341131 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:49.341113 2572 scope.go:117] "RemoveContainer" containerID="c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1" Jul 08 19:52:49.341480 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:52:49.341457 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1\": container with ID starting with c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1 not found: ID does not exist" containerID="c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1" Jul 08 19:52:49.341548 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:49.341492 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1"} err="failed to get container status \"c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1\": rpc error: code = NotFound desc = could not find container \"c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1\": container with ID starting with c70ee7392fe0c2c6e274c983e35c58b660624923242e385004873056a2d29ab1 not found: ID does not exist" Jul 08 19:52:49.360494 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:49.360469 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9bd8bbd4-lzmgh"] Jul 08 19:52:49.369384 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:49.369363 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c9bd8bbd4-lzmgh"] Jul 08 19:52:49.673413 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:52:49.673343 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b838c65f-309a-4939-92c2-b0a10b0b1273" path="/var/lib/kubelet/pods/b838c65f-309a-4939-92c2-b0a10b0b1273/volumes" Jul 08 19:53:32.318692 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.318656 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319090 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3a6113-f959-4f56-949b-8bac13e57cce" containerName="pull" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319110 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3a6113-f959-4f56-949b-8bac13e57cce" containerName="pull" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319124 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" containerName="util" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319132 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" containerName="util" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319143 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" containerName="util" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319152 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" containerName="util" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319180 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a854af9-aeb2-4f8f-a200-edfb4a972e66" containerName="pull" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319189 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a854af9-aeb2-4f8f-a200-edfb4a972e66" containerName="pull" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319202 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" containerName="pull" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319210 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" containerName="pull" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319219 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a854af9-aeb2-4f8f-a200-edfb4a972e66" containerName="util" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319226 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a854af9-aeb2-4f8f-a200-edfb4a972e66" containerName="util" Jul 08 19:53:32.319231 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319236 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b838c65f-309a-4939-92c2-b0a10b0b1273" containerName="console" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319244 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b838c65f-309a-4939-92c2-b0a10b0b1273" containerName="console" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319253 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" containerName="pull" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319261 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" containerName="pull" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319272 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3a6113-f959-4f56-949b-8bac13e57cce" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319280 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3a6113-f959-4f56-949b-8bac13e57cce" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319291 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319299 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319313 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a854af9-aeb2-4f8f-a200-edfb4a972e66" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319321 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a854af9-aeb2-4f8f-a200-edfb4a972e66" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319336 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3a6113-f959-4f56-949b-8bac13e57cce" containerName="util" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319344 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3a6113-f959-4f56-949b-8bac13e57cce" containerName="util" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319354 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319362 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319455 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb1bd3fd-1a07-47e2-8bea-847a456bdaf2" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319467 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a854af9-aeb2-4f8f-a200-edfb4a972e66" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319478 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b838c65f-309a-4939-92c2-b0a10b0b1273" containerName="console" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319489 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc3a6113-f959-4f56-949b-8bac13e57cce" containerName="extract" Jul 08 19:53:32.319849 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.319500 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a66ea7e-0861-43ba-bdb6-f5bfd61573cf" containerName="extract" Jul 08 19:53:32.324232 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.324212 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" Jul 08 19:53:32.326618 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.326599 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xl2pr\"" Jul 08 19:53:32.326717 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.326602 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 19:53:32.326717 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.326661 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 19:53:32.327313 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.327297 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 19:53:32.331972 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.331958 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:53:32.345018 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.344995 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:53:32.427431 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.427396 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3de14616-662d-4c66-8f32-78cf501c7338-config-file\") pod \"limitador-limitador-54d6d988d5-nwfgp\" (UID: \"3de14616-662d-4c66-8f32-78cf501c7338\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" Jul 08 19:53:32.427431 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.427434 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89fmv\" (UniqueName: \"kubernetes.io/projected/3de14616-662d-4c66-8f32-78cf501c7338-kube-api-access-89fmv\") pod \"limitador-limitador-54d6d988d5-nwfgp\" (UID: \"3de14616-662d-4c66-8f32-78cf501c7338\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" Jul 08 19:53:32.528766 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.528733 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3de14616-662d-4c66-8f32-78cf501c7338-config-file\") pod \"limitador-limitador-54d6d988d5-nwfgp\" (UID: \"3de14616-662d-4c66-8f32-78cf501c7338\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" Jul 08 19:53:32.528927 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.528773 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89fmv\" (UniqueName: \"kubernetes.io/projected/3de14616-662d-4c66-8f32-78cf501c7338-kube-api-access-89fmv\") pod \"limitador-limitador-54d6d988d5-nwfgp\" (UID: \"3de14616-662d-4c66-8f32-78cf501c7338\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" Jul 08 19:53:32.529393 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.529372 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3de14616-662d-4c66-8f32-78cf501c7338-config-file\") pod \"limitador-limitador-54d6d988d5-nwfgp\" (UID: \"3de14616-662d-4c66-8f32-78cf501c7338\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" Jul 08 19:53:32.537421 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.537398 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89fmv\" (UniqueName: \"kubernetes.io/projected/3de14616-662d-4c66-8f32-78cf501c7338-kube-api-access-89fmv\") pod \"limitador-limitador-54d6d988d5-nwfgp\" (UID: \"3de14616-662d-4c66-8f32-78cf501c7338\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" Jul 08 19:53:32.635439 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.635372 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" Jul 08 19:53:32.766239 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.766128 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:53:32.768652 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:53:32.768620 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de14616_662d_4c66_8f32_78cf501c7338.slice/crio-e613fd739ba921f35e1879cf932b53c37de5624ea2438e484ff549aa4b27da5d WatchSource:0}: Error finding container e613fd739ba921f35e1879cf932b53c37de5624ea2438e484ff549aa4b27da5d: Status 404 returned error can't find the container with id e613fd739ba921f35e1879cf932b53c37de5624ea2438e484ff549aa4b27da5d Jul 08 19:53:32.770834 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.770818 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:53:32.912633 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.912553 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-4fgvg"] Jul 08 19:53:32.915702 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.915685 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4fgvg" Jul 08 19:53:32.917792 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.917772 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n4hsz\"" Jul 08 19:53:32.921413 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:32.921387 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4fgvg"] Jul 08 19:53:33.033963 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:33.033922 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnp2\" (UniqueName: \"kubernetes.io/projected/14243211-04a9-433f-9358-1bf11bfb26bc-kube-api-access-qxnp2\") pod \"authorino-75d556bd77-4fgvg\" (UID: \"14243211-04a9-433f-9358-1bf11bfb26bc\") " pod="kuadrant-system/authorino-75d556bd77-4fgvg" Jul 08 19:53:33.135038 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:33.134996 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnp2\" (UniqueName: \"kubernetes.io/projected/14243211-04a9-433f-9358-1bf11bfb26bc-kube-api-access-qxnp2\") pod \"authorino-75d556bd77-4fgvg\" (UID: \"14243211-04a9-433f-9358-1bf11bfb26bc\") " pod="kuadrant-system/authorino-75d556bd77-4fgvg" Jul 08 19:53:33.143122 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:33.143088 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnp2\" (UniqueName: \"kubernetes.io/projected/14243211-04a9-433f-9358-1bf11bfb26bc-kube-api-access-qxnp2\") pod \"authorino-75d556bd77-4fgvg\" (UID: \"14243211-04a9-433f-9358-1bf11bfb26bc\") " pod="kuadrant-system/authorino-75d556bd77-4fgvg" Jul 08 19:53:33.225874 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:33.225794 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4fgvg" Jul 08 19:53:33.402240 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:33.401811 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4fgvg"] Jul 08 19:53:33.404302 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:53:33.404257 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14243211_04a9_433f_9358_1bf11bfb26bc.slice/crio-2c15a309a9297bd12fdf130892608e78d7ff18be2deca583edc8e1bbf628a75a WatchSource:0}: Error finding container 2c15a309a9297bd12fdf130892608e78d7ff18be2deca583edc8e1bbf628a75a: Status 404 returned error can't find the container with id 2c15a309a9297bd12fdf130892608e78d7ff18be2deca583edc8e1bbf628a75a Jul 08 19:53:33.555182 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:33.555132 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" event={"ID":"3de14616-662d-4c66-8f32-78cf501c7338","Type":"ContainerStarted","Data":"e613fd739ba921f35e1879cf932b53c37de5624ea2438e484ff549aa4b27da5d"} Jul 08 19:53:33.557011 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:33.556984 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4fgvg" event={"ID":"14243211-04a9-433f-9358-1bf11bfb26bc","Type":"ContainerStarted","Data":"2c15a309a9297bd12fdf130892608e78d7ff18be2deca583edc8e1bbf628a75a"} Jul 08 19:53:37.582109 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:37.582068 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4fgvg" event={"ID":"14243211-04a9-433f-9358-1bf11bfb26bc","Type":"ContainerStarted","Data":"7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b"} Jul 08 19:53:37.583559 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:37.583537 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" event={"ID":"3de14616-662d-4c66-8f32-78cf501c7338","Type":"ContainerStarted","Data":"6ed4b9ed08e3465f3e048a6455a857797c47b51c20cc091867f31ed8031ba145"} Jul 08 19:53:37.583680 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:37.583669 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" Jul 08 19:53:37.606959 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:37.606914 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-4fgvg" podStartSLOduration=1.8586196130000001 podStartE2EDuration="5.60690127s" totalImagesPullingTime="3.748281657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:53:37.605384379 +0000 UTC m=+688.769934781" watchObservedRunningTime="2026-07-08 19:53:37.60690127 +0000 UTC m=+688.771451672" Jul 08 19:53:37.639980 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:37.639932 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" podStartSLOduration=1.256334067 podStartE2EDuration="5.63991605s" totalImagesPullingTime="4.383581983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:53:37.638037613 +0000 UTC m=+688.802588015" watchObservedRunningTime="2026-07-08 19:53:37.63991605 +0000 UTC m=+688.804466452" Jul 08 19:53:48.588690 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:53:48.588659 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nwfgp" Jul 08 19:54:07.123510 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.123478 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-587fc75959-fqxff"] Jul 08 19:54:07.188834 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.188806 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-587fc75959-fqxff"] Jul 08 19:54:07.188982 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.188903 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587fc75959-fqxff" Jul 08 19:54:07.192118 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.192095 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 08 19:54:07.304547 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.304504 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19c7c86f-9ca2-4743-b525-f5b3d976e170-tls-cert\") pod \"authorino-587fc75959-fqxff\" (UID: \"19c7c86f-9ca2-4743-b525-f5b3d976e170\") " pod="kuadrant-system/authorino-587fc75959-fqxff" Jul 08 19:54:07.304700 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.304575 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2r94\" (UniqueName: \"kubernetes.io/projected/19c7c86f-9ca2-4743-b525-f5b3d976e170-kube-api-access-g2r94\") pod \"authorino-587fc75959-fqxff\" (UID: \"19c7c86f-9ca2-4743-b525-f5b3d976e170\") " pod="kuadrant-system/authorino-587fc75959-fqxff" Jul 08 19:54:07.405895 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.405830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19c7c86f-9ca2-4743-b525-f5b3d976e170-tls-cert\") pod \"authorino-587fc75959-fqxff\" (UID: \"19c7c86f-9ca2-4743-b525-f5b3d976e170\") " pod="kuadrant-system/authorino-587fc75959-fqxff" Jul 08 19:54:07.405895 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.405886 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2r94\" (UniqueName: \"kubernetes.io/projected/19c7c86f-9ca2-4743-b525-f5b3d976e170-kube-api-access-g2r94\") pod \"authorino-587fc75959-fqxff\" (UID: \"19c7c86f-9ca2-4743-b525-f5b3d976e170\") " pod="kuadrant-system/authorino-587fc75959-fqxff" Jul 08 19:54:07.408254 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.408229 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19c7c86f-9ca2-4743-b525-f5b3d976e170-tls-cert\") pod \"authorino-587fc75959-fqxff\" (UID: \"19c7c86f-9ca2-4743-b525-f5b3d976e170\") " pod="kuadrant-system/authorino-587fc75959-fqxff" Jul 08 19:54:07.414865 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.414842 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2r94\" (UniqueName: \"kubernetes.io/projected/19c7c86f-9ca2-4743-b525-f5b3d976e170-kube-api-access-g2r94\") pod \"authorino-587fc75959-fqxff\" (UID: \"19c7c86f-9ca2-4743-b525-f5b3d976e170\") " pod="kuadrant-system/authorino-587fc75959-fqxff" Jul 08 19:54:07.498323 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.498301 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587fc75959-fqxff" Jul 08 19:54:07.666986 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.666960 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-587fc75959-fqxff"] Jul 08 19:54:07.669183 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:54:07.669132 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19c7c86f_9ca2_4743_b525_f5b3d976e170.slice/crio-7483836c7eea3b7cd0e1433bfa64ef8b5b9327138236b223b96da59acb3a1320 WatchSource:0}: Error finding container 7483836c7eea3b7cd0e1433bfa64ef8b5b9327138236b223b96da59acb3a1320: Status 404 returned error can't find the container with id 7483836c7eea3b7cd0e1433bfa64ef8b5b9327138236b223b96da59acb3a1320 Jul 08 19:54:07.727897 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:07.727872 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-587fc75959-fqxff" event={"ID":"19c7c86f-9ca2-4743-b525-f5b3d976e170","Type":"ContainerStarted","Data":"7483836c7eea3b7cd0e1433bfa64ef8b5b9327138236b223b96da59acb3a1320"} Jul 08 19:54:08.735093 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:08.735059 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-587fc75959-fqxff" event={"ID":"19c7c86f-9ca2-4743-b525-f5b3d976e170","Type":"ContainerStarted","Data":"df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3"} Jul 08 19:54:08.757722 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:08.757677 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-587fc75959-fqxff" podStartSLOduration=1.233137808 podStartE2EDuration="1.757663439s" totalImagesPullingTime="524.525631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:54:08.75729389 +0000 UTC m=+719.921844292" watchObservedRunningTime="2026-07-08 19:54:08.757663439 +0000 UTC m=+719.922213848" Jul 08 19:54:08.796997 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:08.796963 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4fgvg"] Jul 08 19:54:08.797195 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:08.797172 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-4fgvg" podUID="14243211-04a9-433f-9358-1bf11bfb26bc" containerName="authorino" containerID="cri-o://7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b" gracePeriod=30 Jul 08 19:54:09.048514 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.048491 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4fgvg" Jul 08 19:54:09.119428 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.119400 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxnp2\" (UniqueName: \"kubernetes.io/projected/14243211-04a9-433f-9358-1bf11bfb26bc-kube-api-access-qxnp2\") pod \"14243211-04a9-433f-9358-1bf11bfb26bc\" (UID: \"14243211-04a9-433f-9358-1bf11bfb26bc\") " Jul 08 19:54:09.121400 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.121375 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14243211-04a9-433f-9358-1bf11bfb26bc-kube-api-access-qxnp2" (OuterVolumeSpecName: "kube-api-access-qxnp2") pod "14243211-04a9-433f-9358-1bf11bfb26bc" (UID: "14243211-04a9-433f-9358-1bf11bfb26bc"). InnerVolumeSpecName "kube-api-access-qxnp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:54:09.219994 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.219970 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxnp2\" (UniqueName: \"kubernetes.io/projected/14243211-04a9-433f-9358-1bf11bfb26bc-kube-api-access-qxnp2\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:54:09.740533 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.740500 2572 generic.go:356] "Generic (PLEG): container finished" podID="14243211-04a9-433f-9358-1bf11bfb26bc" containerID="7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b" exitCode=0 Jul 08 19:54:09.740891 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.740535 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4fgvg" Jul 08 19:54:09.740891 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.740580 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4fgvg" event={"ID":"14243211-04a9-433f-9358-1bf11bfb26bc","Type":"ContainerDied","Data":"7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b"} Jul 08 19:54:09.740891 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.740618 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4fgvg" event={"ID":"14243211-04a9-433f-9358-1bf11bfb26bc","Type":"ContainerDied","Data":"2c15a309a9297bd12fdf130892608e78d7ff18be2deca583edc8e1bbf628a75a"} Jul 08 19:54:09.740891 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.740640 2572 scope.go:117] "RemoveContainer" containerID="7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b" Jul 08 19:54:09.748620 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.748605 2572 scope.go:117] "RemoveContainer" containerID="7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b" Jul 08 19:54:09.748854 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:54:09.748839 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b\": container with ID starting with 7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b not found: ID does not exist" containerID="7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b" Jul 08 19:54:09.748925 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.748860 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b"} err="failed to get container status \"7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b\": rpc error: code = NotFound desc = could not find container \"7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b\": container with ID starting with 7df1e2eb9f2daeb5dd7f97321d2589f978dd1e7956d7fc50d7ce887f4678517b not found: ID does not exist" Jul 08 19:54:09.774474 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.774453 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4fgvg"] Jul 08 19:54:09.779050 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:09.779032 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4fgvg"] Jul 08 19:54:11.673109 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:11.673075 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14243211-04a9-433f-9358-1bf11bfb26bc" path="/var/lib/kubelet/pods/14243211-04a9-433f-9358-1bf11bfb26bc/volumes" Jul 08 19:54:51.407763 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:54:51.407729 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:55:07.921100 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:07.921057 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:55:18.299844 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:18.299805 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:55:34.016019 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:34.015946 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:55:42.101279 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:42.101244 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:55:48.802202 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:48.802168 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:55:56.493147 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.493116 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf"] Jul 08 19:55:56.493635 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.493481 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14243211-04a9-433f-9358-1bf11bfb26bc" containerName="authorino" Jul 08 19:55:56.493635 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.493492 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="14243211-04a9-433f-9358-1bf11bfb26bc" containerName="authorino" Jul 08 19:55:56.493635 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.493557 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="14243211-04a9-433f-9358-1bf11bfb26bc" containerName="authorino" Jul 08 19:55:56.496586 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.496571 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.500508 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.500485 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sqmkz\"" Jul 08 19:55:56.500630 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.500527 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 08 19:55:56.500630 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.500574 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 19:55:56.500630 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.500617 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 19:55:56.506832 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.506811 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf"] Jul 08 19:55:56.633023 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.632997 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.633023 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.633025 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.633216 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.633042 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.633216 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.633074 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqbsv\" (UniqueName: \"kubernetes.io/projected/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-kube-api-access-xqbsv\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.633216 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.633104 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.633216 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.633121 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.733741 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.733716 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.733914 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.733745 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.733914 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.733860 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.733914 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.733908 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqbsv\" (UniqueName: \"kubernetes.io/projected/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-kube-api-access-xqbsv\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.734094 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.734026 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.734094 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.734060 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.734094 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.734074 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.734294 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.734196 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.734382 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.734362 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.736002 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.735982 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.736437 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.736422 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.745708 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.745656 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqbsv\" (UniqueName: \"kubernetes.io/projected/bd24a1cc-7f02-4de6-9877-4cd7bde32dda-kube-api-access-xqbsv\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf\" (UID: \"bd24a1cc-7f02-4de6-9877-4cd7bde32dda\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.805310 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.805288 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:55:56.930350 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:56.930320 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf"] Jul 08 19:55:56.932532 ip-10-0-131-22 kubenswrapper[2572]: W0708 19:55:56.932512 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd24a1cc_7f02_4de6_9877_4cd7bde32dda.slice/crio-2a30178356fd71f486f4aa679bd1e1a7584c9f462f4de5b9152c6896a6645109 WatchSource:0}: Error finding container 2a30178356fd71f486f4aa679bd1e1a7584c9f462f4de5b9152c6896a6645109: Status 404 returned error can't find the container with id 2a30178356fd71f486f4aa679bd1e1a7584c9f462f4de5b9152c6896a6645109 Jul 08 19:55:57.262676 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:57.262633 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" event={"ID":"bd24a1cc-7f02-4de6-9877-4cd7bde32dda","Type":"ContainerStarted","Data":"2a30178356fd71f486f4aa679bd1e1a7584c9f462f4de5b9152c6896a6645109"} Jul 08 19:55:58.269235 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:58.269197 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" event={"ID":"bd24a1cc-7f02-4de6-9877-4cd7bde32dda","Type":"ContainerStarted","Data":"69f4d1c808b9bffdf534779ed42cd7fbf3346b0f9aba9afdabe46b338390341c"} Jul 08 19:55:58.288988 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:58.288938 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" podStartSLOduration=1.266598226 podStartE2EDuration="2.288924184s" totalImagesPullingTime="1.022325958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:55:58.288294445 +0000 UTC m=+829.452844850" watchObservedRunningTime="2026-07-08 19:55:58.288924184 +0000 UTC m=+829.453474588" Jul 08 19:55:59.007924 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:55:59.007887 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:56:06.805417 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:56:06.805368 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:56:06.805893 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:56:06.805436 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:56:06.818087 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:56:06.818051 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:56:07.327747 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:56:07.327722 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf" Jul 08 19:57:09.517856 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:09.517825 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 19:57:09.522073 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:09.522053 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 19:57:12.603863 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:12.603827 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-587fc75959-fqxff"] Jul 08 19:57:12.604327 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:12.603999 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-587fc75959-fqxff" podUID="19c7c86f-9ca2-4743-b525-f5b3d976e170" containerName="authorino" containerID="cri-o://df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3" gracePeriod=30 Jul 08 19:57:12.846864 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:12.846845 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587fc75959-fqxff" Jul 08 19:57:12.905241 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:12.905178 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2r94\" (UniqueName: \"kubernetes.io/projected/19c7c86f-9ca2-4743-b525-f5b3d976e170-kube-api-access-g2r94\") pod \"19c7c86f-9ca2-4743-b525-f5b3d976e170\" (UID: \"19c7c86f-9ca2-4743-b525-f5b3d976e170\") " Jul 08 19:57:12.905241 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:12.905235 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19c7c86f-9ca2-4743-b525-f5b3d976e170-tls-cert\") pod \"19c7c86f-9ca2-4743-b525-f5b3d976e170\" (UID: \"19c7c86f-9ca2-4743-b525-f5b3d976e170\") " Jul 08 19:57:12.907127 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:12.907094 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c7c86f-9ca2-4743-b525-f5b3d976e170-kube-api-access-g2r94" (OuterVolumeSpecName: "kube-api-access-g2r94") pod "19c7c86f-9ca2-4743-b525-f5b3d976e170" (UID: "19c7c86f-9ca2-4743-b525-f5b3d976e170"). InnerVolumeSpecName "kube-api-access-g2r94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:57:12.914275 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:12.914249 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c7c86f-9ca2-4743-b525-f5b3d976e170-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "19c7c86f-9ca2-4743-b525-f5b3d976e170" (UID: "19c7c86f-9ca2-4743-b525-f5b3d976e170"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:57:13.006448 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.006420 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2r94\" (UniqueName: \"kubernetes.io/projected/19c7c86f-9ca2-4743-b525-f5b3d976e170-kube-api-access-g2r94\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:57:13.006448 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.006444 2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/19c7c86f-9ca2-4743-b525-f5b3d976e170-tls-cert\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 19:57:13.651002 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.650966 2572 generic.go:356] "Generic (PLEG): container finished" podID="19c7c86f-9ca2-4743-b525-f5b3d976e170" containerID="df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3" exitCode=0 Jul 08 19:57:13.651428 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.651010 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-587fc75959-fqxff" Jul 08 19:57:13.651428 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.651052 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-587fc75959-fqxff" event={"ID":"19c7c86f-9ca2-4743-b525-f5b3d976e170","Type":"ContainerDied","Data":"df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3"} Jul 08 19:57:13.651428 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.651100 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-587fc75959-fqxff" event={"ID":"19c7c86f-9ca2-4743-b525-f5b3d976e170","Type":"ContainerDied","Data":"7483836c7eea3b7cd0e1433bfa64ef8b5b9327138236b223b96da59acb3a1320"} Jul 08 19:57:13.651428 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.651127 2572 scope.go:117] "RemoveContainer" containerID="df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3" Jul 08 19:57:13.660500 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.660484 2572 scope.go:117] "RemoveContainer" containerID="df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3" Jul 08 19:57:13.660806 ip-10-0-131-22 kubenswrapper[2572]: E0708 19:57:13.660789 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3\": container with ID starting with df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3 not found: ID does not exist" containerID="df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3" Jul 08 19:57:13.660850 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.660814 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3"} err="failed to get container status \"df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3\": rpc error: code = NotFound desc = could not find container \"df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3\": container with ID starting with df9551a27b57dea53575e058e6119c2f3fae78af9eeff4210238334bf5a244e3 not found: ID does not exist" Jul 08 19:57:13.681579 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.681552 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-587fc75959-fqxff"] Jul 08 19:57:13.688784 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:13.688767 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-587fc75959-fqxff"] Jul 08 19:57:15.672839 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:15.672810 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c7c86f-9ca2-4743-b525-f5b3d976e170" path="/var/lib/kubelet/pods/19c7c86f-9ca2-4743-b525-f5b3d976e170/volumes" Jul 08 19:57:40.699604 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:40.699569 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:57:50.195781 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:57:50.195743 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:58:01.298753 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:58:01.298719 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:58:11.494968 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:58:11.494934 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:58:21.389510 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:58:21.389480 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:58:47.493510 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:58:47.493475 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:58:59.807613 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:58:59.807581 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:59:32.398592 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:59:32.398555 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:59:42.104984 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:59:42.104946 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:59:49.689922 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:59:49.689891 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 19:59:57.500406 ip-10-0-131-22 kubenswrapper[2572]: I0708 19:59:57.500359 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:00:00.144382 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.144343 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg"] Jul 08 20:00:00.144884 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.144788 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c7c86f-9ca2-4743-b525-f5b3d976e170" containerName="authorino" Jul 08 20:00:00.144884 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.144807 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c7c86f-9ca2-4743-b525-f5b3d976e170" containerName="authorino" Jul 08 20:00:00.145004 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.144909 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c7c86f-9ca2-4743-b525-f5b3d976e170" containerName="authorino" Jul 08 20:00:00.147915 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.147895 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" Jul 08 20:00:00.150351 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.150321 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:00:00.150459 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.150427 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:00:00.151197 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.151184 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:00:00.156322 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.156303 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg"] Jul 08 20:00:00.237397 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.237368 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2ksc\" (UniqueName: \"kubernetes.io/projected/10a1aa3c-359e-4d43-ad14-8a20a3a0c083-kube-api-access-f2ksc\") pod \"maas-api-key-cleanup-29725680-ds9jg\" (UID: \"10a1aa3c-359e-4d43-ad14-8a20a3a0c083\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" Jul 08 20:00:00.338450 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.338396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2ksc\" (UniqueName: \"kubernetes.io/projected/10a1aa3c-359e-4d43-ad14-8a20a3a0c083-kube-api-access-f2ksc\") pod \"maas-api-key-cleanup-29725680-ds9jg\" (UID: \"10a1aa3c-359e-4d43-ad14-8a20a3a0c083\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" Jul 08 20:00:00.346730 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.346700 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2ksc\" (UniqueName: \"kubernetes.io/projected/10a1aa3c-359e-4d43-ad14-8a20a3a0c083-kube-api-access-f2ksc\") pod \"maas-api-key-cleanup-29725680-ds9jg\" (UID: \"10a1aa3c-359e-4d43-ad14-8a20a3a0c083\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" Jul 08 20:00:00.458275 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.458217 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" Jul 08 20:00:00.587690 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.587667 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg"] Jul 08 20:00:00.589618 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:00:00.589587 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a1aa3c_359e_4d43_ad14_8a20a3a0c083.slice/crio-bab155940449c6c40564c24fc3748804c8ec919d1156e7db60fef1cfb9852cc9 WatchSource:0}: Error finding container bab155940449c6c40564c24fc3748804c8ec919d1156e7db60fef1cfb9852cc9: Status 404 returned error can't find the container with id bab155940449c6c40564c24fc3748804c8ec919d1156e7db60fef1cfb9852cc9 Jul 08 20:00:00.591563 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:00.591545 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 20:00:01.470890 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:01.470855 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" event={"ID":"10a1aa3c-359e-4d43-ad14-8a20a3a0c083","Type":"ContainerStarted","Data":"bab155940449c6c40564c24fc3748804c8ec919d1156e7db60fef1cfb9852cc9"} Jul 08 20:00:08.505245 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:08.505212 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:00:08.506697 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:08.506667 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" event={"ID":"10a1aa3c-359e-4d43-ad14-8a20a3a0c083","Type":"ContainerStarted","Data":"fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532"} Jul 08 20:00:08.525301 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:08.525227 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" podStartSLOduration=1.11837189 podStartE2EDuration="8.525212687s" totalImagesPullingTime="7.406840797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:08.523246518 +0000 UTC m=+1079.687796946" watchObservedRunningTime="2026-07-08 20:00:08.525212687 +0000 UTC m=+1079.689763091" Jul 08 20:00:19.313607 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:19.313574 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:00:31.502602 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:31.502568 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:00:41.313102 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:00:41.313069 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:01:00.010826 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:00.010792 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg"] Jul 08 20:01:00.011221 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:00.010973 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" podUID="10a1aa3c-359e-4d43-ad14-8a20a3a0c083" containerName="cleanup" containerID="cri-o://fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532" gracePeriod=30 Jul 08 20:01:03.305879 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:03.305844 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:01:13.408562 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:13.408525 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:01:30.195516 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.195493 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" Jul 08 20:01:30.218260 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.218233 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2ksc\" (UniqueName: \"kubernetes.io/projected/10a1aa3c-359e-4d43-ad14-8a20a3a0c083-kube-api-access-f2ksc\") pod \"10a1aa3c-359e-4d43-ad14-8a20a3a0c083\" (UID: \"10a1aa3c-359e-4d43-ad14-8a20a3a0c083\") " Jul 08 20:01:30.220243 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.220214 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a1aa3c-359e-4d43-ad14-8a20a3a0c083-kube-api-access-f2ksc" (OuterVolumeSpecName: "kube-api-access-f2ksc") pod "10a1aa3c-359e-4d43-ad14-8a20a3a0c083" (UID: "10a1aa3c-359e-4d43-ad14-8a20a3a0c083"). InnerVolumeSpecName "kube-api-access-f2ksc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:01:30.319010 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.318954 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2ksc\" (UniqueName: \"kubernetes.io/projected/10a1aa3c-359e-4d43-ad14-8a20a3a0c083-kube-api-access-f2ksc\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:01:30.904682 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.904646 2572 generic.go:356] "Generic (PLEG): container finished" podID="10a1aa3c-359e-4d43-ad14-8a20a3a0c083" containerID="fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532" exitCode=137 Jul 08 20:01:30.904682 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.904684 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" event={"ID":"10a1aa3c-359e-4d43-ad14-8a20a3a0c083","Type":"ContainerDied","Data":"fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532"} Jul 08 20:01:30.904910 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.904705 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" Jul 08 20:01:30.904910 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.904728 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg" event={"ID":"10a1aa3c-359e-4d43-ad14-8a20a3a0c083","Type":"ContainerDied","Data":"bab155940449c6c40564c24fc3748804c8ec919d1156e7db60fef1cfb9852cc9"} Jul 08 20:01:30.904910 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.904749 2572 scope.go:117] "RemoveContainer" containerID="fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532" Jul 08 20:01:30.913654 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.913637 2572 scope.go:117] "RemoveContainer" containerID="fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532" Jul 08 20:01:30.913896 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:01:30.913875 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532\": container with ID starting with fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532 not found: ID does not exist" containerID="fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532" Jul 08 20:01:30.913938 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.913904 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532"} err="failed to get container status \"fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532\": rpc error: code = NotFound desc = could not find container \"fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532\": container with ID starting with fa4b24d9a2307e0377c9766b6cdfef7d472afdfa27e751a73d996434b08e8532 not found: ID does not exist" Jul 08 20:01:30.927621 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.927599 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg"] Jul 08 20:01:30.934281 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:30.934261 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725680-ds9jg"] Jul 08 20:01:31.674616 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:31.674588 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a1aa3c-359e-4d43-ad14-8a20a3a0c083" path="/var/lib/kubelet/pods/10a1aa3c-359e-4d43-ad14-8a20a3a0c083/volumes" Jul 08 20:01:35.606745 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:35.606711 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:01:47.010770 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:01:47.010736 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:02:03.510752 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:02:03.510716 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:02:09.551309 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:02:09.550812 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:02:09.559693 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:02:09.559665 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:02:14.905552 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:02:14.905524 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:02:24.522773 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:02:24.522741 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:02:34.503179 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:02:34.503116 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:02:44.200437 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:02:44.200406 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:04:30.419037 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:04:30.418964 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:04:43.213653 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:04:43.213621 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:04:51.322197 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:04:51.322143 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:05:02.293060 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:05:02.293024 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:05:15.106560 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:05:15.106515 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:05:25.405540 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:05:25.405503 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:05:33.408244 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:05:33.408209 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:05:43.508602 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:05:43.508566 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:05:55.334631 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:05:55.334597 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:06:03.900613 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:06:03.900576 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:06:14.706567 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:06:14.706535 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:06:36.440111 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:06:36.440076 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:06:45.751815 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:06:45.751784 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:06:56.130696 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:06:56.130662 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:07:06.004955 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:07:06.004918 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:07:09.585899 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:07:09.585873 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:07:09.590512 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:07:09.590482 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:07:12.802992 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:07:12.802955 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:07:29.016809 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:07:29.016771 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:07:38.908545 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:07:38.908467 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:07:48.718668 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:07:48.718631 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:07:59.402916 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:07:59.402880 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:08:09.103540 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:08:09.103505 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:08:25.818351 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:08:25.818319 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:08:35.312175 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:08:35.312122 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:08:44.912312 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:08:44.912277 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:08:55.108043 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:08:55.108003 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:09:05.097927 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:09:05.097891 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:09:18.250658 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:09:18.250627 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:09:26.310293 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:09:26.310265 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:09:39.008733 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:09:39.008702 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:09:50.294102 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:09:50.294068 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:10:04.028170 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:04.028134 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:10:18.107027 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:18.106993 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:10:27.194127 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:27.194091 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:10:37.500348 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.500315 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp"] Jul 08 20:10:37.500818 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.500627 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a1aa3c-359e-4d43-ad14-8a20a3a0c083" containerName="cleanup" Jul 08 20:10:37.500818 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.500639 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a1aa3c-359e-4d43-ad14-8a20a3a0c083" containerName="cleanup" Jul 08 20:10:37.500818 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.500725 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a1aa3c-359e-4d43-ad14-8a20a3a0c083" containerName="cleanup" Jul 08 20:10:37.503757 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.503741 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.506346 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.506320 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5dd1d121-serving-cert\"" Jul 08 20:10:37.506598 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.506582 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:10:37.506710 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.506692 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:10:37.507254 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.507237 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:10:37.515830 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.515810 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp"] Jul 08 20:10:37.591222 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.591198 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a978def0-668b-4f1b-894a-16c1cf437fff-openshift-service-ca\") pod \"maas-api-e2e-ait-5dd1d121-7fff485975-wpblp\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.591371 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.591257 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a978def0-668b-4f1b-894a-16c1cf437fff-maas-api-tls\") pod \"maas-api-e2e-ait-5dd1d121-7fff485975-wpblp\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.591371 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.591338 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f98j5\" (UniqueName: \"kubernetes.io/projected/a978def0-668b-4f1b-894a-16c1cf437fff-kube-api-access-f98j5\") pod \"maas-api-e2e-ait-5dd1d121-7fff485975-wpblp\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.691852 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.691826 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a978def0-668b-4f1b-894a-16c1cf437fff-openshift-service-ca\") pod \"maas-api-e2e-ait-5dd1d121-7fff485975-wpblp\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.691980 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.691866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a978def0-668b-4f1b-894a-16c1cf437fff-maas-api-tls\") pod \"maas-api-e2e-ait-5dd1d121-7fff485975-wpblp\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.691980 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.691896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f98j5\" (UniqueName: \"kubernetes.io/projected/a978def0-668b-4f1b-894a-16c1cf437fff-kube-api-access-f98j5\") pod \"maas-api-e2e-ait-5dd1d121-7fff485975-wpblp\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.692438 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.692419 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a978def0-668b-4f1b-894a-16c1cf437fff-openshift-service-ca\") pod \"maas-api-e2e-ait-5dd1d121-7fff485975-wpblp\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.694355 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.694336 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a978def0-668b-4f1b-894a-16c1cf437fff-maas-api-tls\") pod \"maas-api-e2e-ait-5dd1d121-7fff485975-wpblp\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.700490 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.700470 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f98j5\" (UniqueName: \"kubernetes.io/projected/a978def0-668b-4f1b-894a-16c1cf437fff-kube-api-access-f98j5\") pod \"maas-api-e2e-ait-5dd1d121-7fff485975-wpblp\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.815310 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.815291 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:37.901186 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.901144 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:10:37.950757 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.950730 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp"] Jul 08 20:10:37.953313 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:10:37.953288 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda978def0_668b_4f1b_894a_16c1cf437fff.slice/crio-45afc5d62baa7bfde0c6c5b913c4869bc2f34157c1bd571428e69091b010abe5 WatchSource:0}: Error finding container 45afc5d62baa7bfde0c6c5b913c4869bc2f34157c1bd571428e69091b010abe5: Status 404 returned error can't find the container with id 45afc5d62baa7bfde0c6c5b913c4869bc2f34157c1bd571428e69091b010abe5 Jul 08 20:10:37.954673 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:37.954658 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 20:10:38.565798 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:38.565761 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" event={"ID":"a978def0-668b-4f1b-894a-16c1cf437fff","Type":"ContainerStarted","Data":"45afc5d62baa7bfde0c6c5b913c4869bc2f34157c1bd571428e69091b010abe5"} Jul 08 20:10:40.584292 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:40.584261 2572 generic.go:356] "Generic (PLEG): container finished" podID="a978def0-668b-4f1b-894a-16c1cf437fff" containerID="77ea0d25bc72f352744df872f6aefc880ec4f56018b31a0c1c6346f65742ee01" exitCode=1 Jul 08 20:10:40.584662 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:40.584319 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" event={"ID":"a978def0-668b-4f1b-894a-16c1cf437fff","Type":"ContainerDied","Data":"77ea0d25bc72f352744df872f6aefc880ec4f56018b31a0c1c6346f65742ee01"} Jul 08 20:10:40.584662 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:40.584547 2572 scope.go:117] "RemoveContainer" containerID="77ea0d25bc72f352744df872f6aefc880ec4f56018b31a0c1c6346f65742ee01" Jul 08 20:10:41.591371 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:41.591294 2572 generic.go:356] "Generic (PLEG): container finished" podID="a978def0-668b-4f1b-894a-16c1cf437fff" containerID="c0f97eea3172aab98b2d4e8be4fa19902a6ffe9cc21f9e0db1ed1fbac6e7eed3" exitCode=1 Jul 08 20:10:41.591724 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:41.591380 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" event={"ID":"a978def0-668b-4f1b-894a-16c1cf437fff","Type":"ContainerDied","Data":"c0f97eea3172aab98b2d4e8be4fa19902a6ffe9cc21f9e0db1ed1fbac6e7eed3"} Jul 08 20:10:41.591724 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:41.591438 2572 scope.go:117] "RemoveContainer" containerID="77ea0d25bc72f352744df872f6aefc880ec4f56018b31a0c1c6346f65742ee01" Jul 08 20:10:41.591724 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:41.591622 2572 scope.go:117] "RemoveContainer" containerID="c0f97eea3172aab98b2d4e8be4fa19902a6ffe9cc21f9e0db1ed1fbac6e7eed3" Jul 08 20:10:41.591858 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:41.591829 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-5dd1d121-7fff485975-wpblp_odh-ai-gateway-infra(a978def0-668b-4f1b-894a-16c1cf437fff)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" podUID="a978def0-668b-4f1b-894a-16c1cf437fff" Jul 08 20:10:42.598110 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:42.598083 2572 scope.go:117] "RemoveContainer" containerID="c0f97eea3172aab98b2d4e8be4fa19902a6ffe9cc21f9e0db1ed1fbac6e7eed3" Jul 08 20:10:42.598505 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:42.598287 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-5dd1d121-7fff485975-wpblp_odh-ai-gateway-infra(a978def0-668b-4f1b-894a-16c1cf437fff)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" podUID="a978def0-668b-4f1b-894a-16c1cf437fff" Jul 08 20:10:42.816461 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:42.816431 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:42.881942 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:42.881874 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp"] Jul 08 20:10:43.602519 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:43.602489 2572 scope.go:117] "RemoveContainer" containerID="c0f97eea3172aab98b2d4e8be4fa19902a6ffe9cc21f9e0db1ed1fbac6e7eed3" Jul 08 20:10:43.602897 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:43.602660 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-5dd1d121-7fff485975-wpblp_odh-ai-gateway-infra(a978def0-668b-4f1b-894a-16c1cf437fff)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" podUID="a978def0-668b-4f1b-894a-16c1cf437fff" Jul 08 20:10:43.634797 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:43.634762 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-serving-cert: secret "maas-api-e2e-ait-5dd1d121-serving-cert" not found Jul 08 20:10:43.634963 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:43.634831 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a978def0-668b-4f1b-894a-16c1cf437fff-maas-api-tls podName:a978def0-668b-4f1b-894a-16c1cf437fff nodeName:}" failed. No retries permitted until 2026-07-08 20:10:44.134811853 +0000 UTC m=+1715.299362238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a978def0-668b-4f1b-894a-16c1cf437fff-maas-api-tls") pod "maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" (UID: "a978def0-668b-4f1b-894a-16c1cf437fff") : secret "maas-api-e2e-ait-5dd1d121-serving-cert" not found Jul 08 20:10:44.139314 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:44.139283 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-serving-cert: secret "maas-api-e2e-ait-5dd1d121-serving-cert" not found Jul 08 20:10:44.139473 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:44.139346 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a978def0-668b-4f1b-894a-16c1cf437fff-maas-api-tls podName:a978def0-668b-4f1b-894a-16c1cf437fff nodeName:}" failed. No retries permitted until 2026-07-08 20:10:45.139331129 +0000 UTC m=+1716.303881511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a978def0-668b-4f1b-894a-16c1cf437fff-maas-api-tls") pod "maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" (UID: "a978def0-668b-4f1b-894a-16c1cf437fff") : secret "maas-api-e2e-ait-5dd1d121-serving-cert" not found Jul 08 20:10:44.736401 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:44.736379 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:44.743691 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:44.743671 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f98j5\" (UniqueName: \"kubernetes.io/projected/a978def0-668b-4f1b-894a-16c1cf437fff-kube-api-access-f98j5\") pod \"a978def0-668b-4f1b-894a-16c1cf437fff\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " Jul 08 20:10:44.743787 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:44.743761 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a978def0-668b-4f1b-894a-16c1cf437fff-openshift-service-ca\") pod \"a978def0-668b-4f1b-894a-16c1cf437fff\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " Jul 08 20:10:44.743826 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:44.743788 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a978def0-668b-4f1b-894a-16c1cf437fff-maas-api-tls\") pod \"a978def0-668b-4f1b-894a-16c1cf437fff\" (UID: \"a978def0-668b-4f1b-894a-16c1cf437fff\") " Jul 08 20:10:44.744148 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:44.744121 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a978def0-668b-4f1b-894a-16c1cf437fff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a978def0-668b-4f1b-894a-16c1cf437fff" (UID: "a978def0-668b-4f1b-894a-16c1cf437fff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:10:44.745611 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:44.745578 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a978def0-668b-4f1b-894a-16c1cf437fff-kube-api-access-f98j5" (OuterVolumeSpecName: "kube-api-access-f98j5") pod "a978def0-668b-4f1b-894a-16c1cf437fff" (UID: "a978def0-668b-4f1b-894a-16c1cf437fff"). InnerVolumeSpecName "kube-api-access-f98j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:10:44.745771 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:44.745752 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a978def0-668b-4f1b-894a-16c1cf437fff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a978def0-668b-4f1b-894a-16c1cf437fff" (UID: "a978def0-668b-4f1b-894a-16c1cf437fff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:10:44.845383 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:44.845354 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f98j5\" (UniqueName: \"kubernetes.io/projected/a978def0-668b-4f1b-894a-16c1cf437fff-kube-api-access-f98j5\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:10:44.845383 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:44.845378 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a978def0-668b-4f1b-894a-16c1cf437fff-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:10:44.845533 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:44.845389 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a978def0-668b-4f1b-894a-16c1cf437fff-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:10:45.618776 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:45.618746 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" Jul 08 20:10:45.618776 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:45.618767 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp" event={"ID":"a978def0-668b-4f1b-894a-16c1cf437fff","Type":"ContainerDied","Data":"45afc5d62baa7bfde0c6c5b913c4869bc2f34157c1bd571428e69091b010abe5"} Jul 08 20:10:45.618973 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:45.618849 2572 scope.go:117] "RemoveContainer" containerID="c0f97eea3172aab98b2d4e8be4fa19902a6ffe9cc21f9e0db1ed1fbac6e7eed3" Jul 08 20:10:45.641269 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:45.641241 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp"] Jul 08 20:10:45.646897 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:45.646877 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5dd1d121-7fff485975-wpblp"] Jul 08 20:10:45.673115 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:45.673092 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a978def0-668b-4f1b-894a-16c1cf437fff" path="/var/lib/kubelet/pods/a978def0-668b-4f1b-894a-16c1cf437fff/volumes" Jul 08 20:10:50.262867 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.262832 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj"] Jul 08 20:10:50.263223 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.263168 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a978def0-668b-4f1b-894a-16c1cf437fff" containerName="maas-api" Jul 08 20:10:50.263223 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.263185 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a978def0-668b-4f1b-894a-16c1cf437fff" containerName="maas-api" Jul 08 20:10:50.263223 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.263199 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a978def0-668b-4f1b-894a-16c1cf437fff" containerName="maas-api" Jul 08 20:10:50.263223 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.263205 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a978def0-668b-4f1b-894a-16c1cf437fff" containerName="maas-api" Jul 08 20:10:50.263364 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.263264 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a978def0-668b-4f1b-894a-16c1cf437fff" containerName="maas-api" Jul 08 20:10:50.267478 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.267457 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:50.272024 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.272004 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:10:50.272317 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.272303 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:10:50.272730 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.272711 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fedc6cb5-serving-cert\"" Jul 08 20:10:50.272822 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.272750 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:10:50.287369 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.287346 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2l94\" (UniqueName: \"kubernetes.io/projected/c6d8bbfc-876f-43dc-9921-7318fef899c7-kube-api-access-s2l94\") pod \"maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:50.287469 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.287438 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d8bbfc-876f-43dc-9921-7318fef899c7-maas-api-tls\") pod \"maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:50.287538 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.287468 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8bbfc-876f-43dc-9921-7318fef899c7-openshift-service-ca\") pod \"maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:50.288296 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.288272 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj"] Jul 08 20:10:50.388265 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.388236 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2l94\" (UniqueName: \"kubernetes.io/projected/c6d8bbfc-876f-43dc-9921-7318fef899c7-kube-api-access-s2l94\") pod \"maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:50.388391 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.388300 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d8bbfc-876f-43dc-9921-7318fef899c7-maas-api-tls\") pod \"maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:50.388391 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.388330 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8bbfc-876f-43dc-9921-7318fef899c7-openshift-service-ca\") pod \"maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:50.388487 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:50.388436 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-serving-cert: secret "maas-api-e2e-ait-fedc6cb5-serving-cert" not found Jul 08 20:10:50.388526 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:50.388503 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6d8bbfc-876f-43dc-9921-7318fef899c7-maas-api-tls podName:c6d8bbfc-876f-43dc-9921-7318fef899c7 nodeName:}" failed. No retries permitted until 2026-07-08 20:10:50.888485397 +0000 UTC m=+1722.053035786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6d8bbfc-876f-43dc-9921-7318fef899c7-maas-api-tls") pod "maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" (UID: "c6d8bbfc-876f-43dc-9921-7318fef899c7") : secret "maas-api-e2e-ait-fedc6cb5-serving-cert" not found Jul 08 20:10:50.388863 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.388846 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8bbfc-876f-43dc-9921-7318fef899c7-openshift-service-ca\") pod \"maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:50.419263 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.419238 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2l94\" (UniqueName: \"kubernetes.io/projected/c6d8bbfc-876f-43dc-9921-7318fef899c7-kube-api-access-s2l94\") pod \"maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:50.891997 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.891956 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d8bbfc-876f-43dc-9921-7318fef899c7-maas-api-tls\") pod \"maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:50.894471 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:50.894448 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d8bbfc-876f-43dc-9921-7318fef899c7-maas-api-tls\") pod \"maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:51.178256 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:51.178186 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:51.315964 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:51.315940 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj"] Jul 08 20:10:51.318000 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:10:51.317966 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d8bbfc_876f_43dc_9921_7318fef899c7.slice/crio-e2f27302ffcd7fd01d0a15fc9b517e5db46592d142fb5acaeaedca600f0988f6 WatchSource:0}: Error finding container e2f27302ffcd7fd01d0a15fc9b517e5db46592d142fb5acaeaedca600f0988f6: Status 404 returned error can't find the container with id e2f27302ffcd7fd01d0a15fc9b517e5db46592d142fb5acaeaedca600f0988f6 Jul 08 20:10:51.649140 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:51.649118 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" event={"ID":"c6d8bbfc-876f-43dc-9921-7318fef899c7","Type":"ContainerStarted","Data":"e2f27302ffcd7fd01d0a15fc9b517e5db46592d142fb5acaeaedca600f0988f6"} Jul 08 20:10:52.655614 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:52.655526 2572 generic.go:356] "Generic (PLEG): container finished" podID="c6d8bbfc-876f-43dc-9921-7318fef899c7" containerID="fef917c54a51094649f2fc80a187c09fda719c83e45bae8544f2be1f37ffe891" exitCode=1 Jul 08 20:10:52.655968 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:52.655609 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" event={"ID":"c6d8bbfc-876f-43dc-9921-7318fef899c7","Type":"ContainerDied","Data":"fef917c54a51094649f2fc80a187c09fda719c83e45bae8544f2be1f37ffe891"} Jul 08 20:10:52.655968 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:52.655859 2572 scope.go:117] "RemoveContainer" containerID="fef917c54a51094649f2fc80a187c09fda719c83e45bae8544f2be1f37ffe891" Jul 08 20:10:53.662090 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:53.662004 2572 generic.go:356] "Generic (PLEG): container finished" podID="c6d8bbfc-876f-43dc-9921-7318fef899c7" containerID="89a30763f66b53798edbc99fb45c73c9934bb194970e1e52591b95b467ef9389" exitCode=1 Jul 08 20:10:53.662477 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:53.662095 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" event={"ID":"c6d8bbfc-876f-43dc-9921-7318fef899c7","Type":"ContainerDied","Data":"89a30763f66b53798edbc99fb45c73c9934bb194970e1e52591b95b467ef9389"} Jul 08 20:10:53.662477 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:53.662151 2572 scope.go:117] "RemoveContainer" containerID="fef917c54a51094649f2fc80a187c09fda719c83e45bae8544f2be1f37ffe891" Jul 08 20:10:53.662477 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:53.662289 2572 scope.go:117] "RemoveContainer" containerID="89a30763f66b53798edbc99fb45c73c9934bb194970e1e52591b95b467ef9389" Jul 08 20:10:53.662646 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:53.662497 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj_odh-ai-gateway-infra(c6d8bbfc-876f-43dc-9921-7318fef899c7)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" podUID="c6d8bbfc-876f-43dc-9921-7318fef899c7" Jul 08 20:10:54.669847 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:54.669820 2572 scope.go:117] "RemoveContainer" containerID="89a30763f66b53798edbc99fb45c73c9934bb194970e1e52591b95b467ef9389" Jul 08 20:10:54.670257 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:10:54.669957 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj_odh-ai-gateway-infra(c6d8bbfc-876f-43dc-9921-7318fef899c7)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" podUID="c6d8bbfc-876f-43dc-9921-7318fef899c7" Jul 08 20:10:55.532872 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.532842 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj"] Jul 08 20:10:55.804700 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.804675 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:55.806307 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.806285 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:10:55.831937 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.831910 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d8bbfc-876f-43dc-9921-7318fef899c7-maas-api-tls\") pod \"c6d8bbfc-876f-43dc-9921-7318fef899c7\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " Jul 08 20:10:55.832064 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.831978 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8bbfc-876f-43dc-9921-7318fef899c7-openshift-service-ca\") pod \"c6d8bbfc-876f-43dc-9921-7318fef899c7\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " Jul 08 20:10:55.832064 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.832051 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2l94\" (UniqueName: \"kubernetes.io/projected/c6d8bbfc-876f-43dc-9921-7318fef899c7-kube-api-access-s2l94\") pod \"c6d8bbfc-876f-43dc-9921-7318fef899c7\" (UID: \"c6d8bbfc-876f-43dc-9921-7318fef899c7\") " Jul 08 20:10:55.832374 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.832348 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d8bbfc-876f-43dc-9921-7318fef899c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c6d8bbfc-876f-43dc-9921-7318fef899c7" (UID: "c6d8bbfc-876f-43dc-9921-7318fef899c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:10:55.834119 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.834095 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d8bbfc-876f-43dc-9921-7318fef899c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c6d8bbfc-876f-43dc-9921-7318fef899c7" (UID: "c6d8bbfc-876f-43dc-9921-7318fef899c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:10:55.834547 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.834522 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d8bbfc-876f-43dc-9921-7318fef899c7-kube-api-access-s2l94" (OuterVolumeSpecName: "kube-api-access-s2l94") pod "c6d8bbfc-876f-43dc-9921-7318fef899c7" (UID: "c6d8bbfc-876f-43dc-9921-7318fef899c7"). InnerVolumeSpecName "kube-api-access-s2l94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:10:55.932912 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.932889 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2l94\" (UniqueName: \"kubernetes.io/projected/c6d8bbfc-876f-43dc-9921-7318fef899c7-kube-api-access-s2l94\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:10:55.933009 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.932914 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6d8bbfc-876f-43dc-9921-7318fef899c7-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:10:55.933009 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:55.932928 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8bbfc-876f-43dc-9921-7318fef899c7-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:10:56.687899 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:56.687860 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" event={"ID":"c6d8bbfc-876f-43dc-9921-7318fef899c7","Type":"ContainerDied","Data":"e2f27302ffcd7fd01d0a15fc9b517e5db46592d142fb5acaeaedca600f0988f6"} Jul 08 20:10:56.688058 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:56.687912 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj" Jul 08 20:10:56.688058 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:56.687960 2572 scope.go:117] "RemoveContainer" containerID="89a30763f66b53798edbc99fb45c73c9934bb194970e1e52591b95b467ef9389" Jul 08 20:10:56.709830 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:56.709808 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj"] Jul 08 20:10:56.717201 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:56.717176 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fedc6cb5-576bf6f7b5-gtnjj"] Jul 08 20:10:57.673060 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:10:57.673026 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d8bbfc-876f-43dc-9921-7318fef899c7" path="/var/lib/kubelet/pods/c6d8bbfc-876f-43dc-9921-7318fef899c7/volumes" Jul 08 20:11:04.534933 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:04.534901 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:11:08.804618 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.804587 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5"] Jul 08 20:11:08.805047 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.805033 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d8bbfc-876f-43dc-9921-7318fef899c7" containerName="maas-api" Jul 08 20:11:08.805103 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.805050 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d8bbfc-876f-43dc-9921-7318fef899c7" containerName="maas-api" Jul 08 20:11:08.805103 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.805066 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d8bbfc-876f-43dc-9921-7318fef899c7" containerName="maas-api" Jul 08 20:11:08.805103 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.805074 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d8bbfc-876f-43dc-9921-7318fef899c7" containerName="maas-api" Jul 08 20:11:08.805219 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.805191 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a978def0-668b-4f1b-894a-16c1cf437fff" containerName="maas-api" Jul 08 20:11:08.805219 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.805203 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d8bbfc-876f-43dc-9921-7318fef899c7" containerName="maas-api" Jul 08 20:11:08.809756 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.809735 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:08.811980 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.811956 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e1c198eb-serving-cert\"" Jul 08 20:11:08.812123 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.812070 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:11:08.812373 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.812355 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:11:08.812889 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.812868 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:11:08.818908 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.818840 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5"] Jul 08 20:11:08.928923 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.928893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fc569c-559e-44d1-88d8-6789b4302ba6-maas-api-tls\") pod \"maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:08.929080 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.928947 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fc569c-559e-44d1-88d8-6789b4302ba6-openshift-service-ca\") pod \"maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:08.929080 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:08.929007 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d676v\" (UniqueName: \"kubernetes.io/projected/52fc569c-559e-44d1-88d8-6789b4302ba6-kube-api-access-d676v\") pod \"maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:09.029743 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:09.029708 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d676v\" (UniqueName: \"kubernetes.io/projected/52fc569c-559e-44d1-88d8-6789b4302ba6-kube-api-access-d676v\") pod \"maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:09.029903 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:09.029756 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fc569c-559e-44d1-88d8-6789b4302ba6-maas-api-tls\") pod \"maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:09.029903 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:09.029802 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fc569c-559e-44d1-88d8-6789b4302ba6-openshift-service-ca\") pod \"maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:09.029991 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:11:09.029925 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-serving-cert: secret "maas-api-e2e-derive-e1c198eb-serving-cert" not found Jul 08 20:11:09.030027 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:11:09.029994 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52fc569c-559e-44d1-88d8-6789b4302ba6-maas-api-tls podName:52fc569c-559e-44d1-88d8-6789b4302ba6 nodeName:}" failed. No retries permitted until 2026-07-08 20:11:09.529971592 +0000 UTC m=+1740.694521985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52fc569c-559e-44d1-88d8-6789b4302ba6-maas-api-tls") pod "maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" (UID: "52fc569c-559e-44d1-88d8-6789b4302ba6") : secret "maas-api-e2e-derive-e1c198eb-serving-cert" not found Jul 08 20:11:09.030472 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:09.030455 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fc569c-559e-44d1-88d8-6789b4302ba6-openshift-service-ca\") pod \"maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:09.038012 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:09.037990 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d676v\" (UniqueName: \"kubernetes.io/projected/52fc569c-559e-44d1-88d8-6789b4302ba6-kube-api-access-d676v\") pod \"maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:09.534613 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:09.534576 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fc569c-559e-44d1-88d8-6789b4302ba6-maas-api-tls\") pod \"maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:09.537061 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:09.537027 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fc569c-559e-44d1-88d8-6789b4302ba6-maas-api-tls\") pod \"maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:09.724322 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:09.724297 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:11:09.732380 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:09.732360 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:09.861582 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:09.861546 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5"] Jul 08 20:11:09.864513 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:11:09.864487 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fc569c_559e_44d1_88d8_6789b4302ba6.slice/crio-b6e97679d9ee496c23244f20d905811deda91552731587f79e9ba688f56ab8f9 WatchSource:0}: Error finding container b6e97679d9ee496c23244f20d905811deda91552731587f79e9ba688f56ab8f9: Status 404 returned error can't find the container with id b6e97679d9ee496c23244f20d905811deda91552731587f79e9ba688f56ab8f9 Jul 08 20:11:10.755306 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:10.755274 2572 generic.go:356] "Generic (PLEG): container finished" podID="52fc569c-559e-44d1-88d8-6789b4302ba6" containerID="7b0da87511fbdaa64566f1ad01f7b9b621db9410feb544bcb7c08e8540967bf4" exitCode=1 Jul 08 20:11:10.755475 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:10.755358 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" event={"ID":"52fc569c-559e-44d1-88d8-6789b4302ba6","Type":"ContainerDied","Data":"7b0da87511fbdaa64566f1ad01f7b9b621db9410feb544bcb7c08e8540967bf4"} Jul 08 20:11:10.755475 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:10.755410 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" event={"ID":"52fc569c-559e-44d1-88d8-6789b4302ba6","Type":"ContainerStarted","Data":"b6e97679d9ee496c23244f20d905811deda91552731587f79e9ba688f56ab8f9"} Jul 08 20:11:10.755600 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:10.755588 2572 scope.go:117] "RemoveContainer" containerID="7b0da87511fbdaa64566f1ad01f7b9b621db9410feb544bcb7c08e8540967bf4" Jul 08 20:11:11.762949 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:11.762842 2572 generic.go:356] "Generic (PLEG): container finished" podID="52fc569c-559e-44d1-88d8-6789b4302ba6" containerID="55720b7f55fc3a73d2b50c994cbf79eba475dff1f153bd2a97ca58fb89b71c93" exitCode=1 Jul 08 20:11:11.762949 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:11.762914 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" event={"ID":"52fc569c-559e-44d1-88d8-6789b4302ba6","Type":"ContainerDied","Data":"55720b7f55fc3a73d2b50c994cbf79eba475dff1f153bd2a97ca58fb89b71c93"} Jul 08 20:11:11.763395 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:11.762971 2572 scope.go:117] "RemoveContainer" containerID="7b0da87511fbdaa64566f1ad01f7b9b621db9410feb544bcb7c08e8540967bf4" Jul 08 20:11:11.763395 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:11.763220 2572 scope.go:117] "RemoveContainer" containerID="55720b7f55fc3a73d2b50c994cbf79eba475dff1f153bd2a97ca58fb89b71c93" Jul 08 20:11:11.763472 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:11:11.763405 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5_odh-ai-gateway-infra(52fc569c-559e-44d1-88d8-6789b4302ba6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" podUID="52fc569c-559e-44d1-88d8-6789b4302ba6" Jul 08 20:11:12.769345 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:12.769317 2572 scope.go:117] "RemoveContainer" containerID="55720b7f55fc3a73d2b50c994cbf79eba475dff1f153bd2a97ca58fb89b71c93" Jul 08 20:11:12.769700 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:11:12.769485 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5_odh-ai-gateway-infra(52fc569c-559e-44d1-88d8-6789b4302ba6)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" podUID="52fc569c-559e-44d1-88d8-6789b4302ba6" Jul 08 20:11:13.593532 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:13.593499 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5"] Jul 08 20:11:13.905514 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:13.905487 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:13.969692 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:13.969669 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fc569c-559e-44d1-88d8-6789b4302ba6-openshift-service-ca\") pod \"52fc569c-559e-44d1-88d8-6789b4302ba6\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " Jul 08 20:11:13.969825 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:13.969724 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fc569c-559e-44d1-88d8-6789b4302ba6-maas-api-tls\") pod \"52fc569c-559e-44d1-88d8-6789b4302ba6\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " Jul 08 20:11:13.969825 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:13.969751 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d676v\" (UniqueName: \"kubernetes.io/projected/52fc569c-559e-44d1-88d8-6789b4302ba6-kube-api-access-d676v\") pod \"52fc569c-559e-44d1-88d8-6789b4302ba6\" (UID: \"52fc569c-559e-44d1-88d8-6789b4302ba6\") " Jul 08 20:11:13.970053 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:13.970025 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fc569c-559e-44d1-88d8-6789b4302ba6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52fc569c-559e-44d1-88d8-6789b4302ba6" (UID: "52fc569c-559e-44d1-88d8-6789b4302ba6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:11:13.971716 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:13.971689 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52fc569c-559e-44d1-88d8-6789b4302ba6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52fc569c-559e-44d1-88d8-6789b4302ba6" (UID: "52fc569c-559e-44d1-88d8-6789b4302ba6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:11:13.971797 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:13.971744 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fc569c-559e-44d1-88d8-6789b4302ba6-kube-api-access-d676v" (OuterVolumeSpecName: "kube-api-access-d676v") pod "52fc569c-559e-44d1-88d8-6789b4302ba6" (UID: "52fc569c-559e-44d1-88d8-6789b4302ba6"). InnerVolumeSpecName "kube-api-access-d676v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:11:14.071242 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:14.071217 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52fc569c-559e-44d1-88d8-6789b4302ba6-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:11:14.071242 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:14.071239 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52fc569c-559e-44d1-88d8-6789b4302ba6-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:11:14.071430 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:14.071252 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d676v\" (UniqueName: \"kubernetes.io/projected/52fc569c-559e-44d1-88d8-6789b4302ba6-kube-api-access-d676v\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:11:14.780318 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:14.780289 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" Jul 08 20:11:14.780318 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:14.780295 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5" event={"ID":"52fc569c-559e-44d1-88d8-6789b4302ba6","Type":"ContainerDied","Data":"b6e97679d9ee496c23244f20d905811deda91552731587f79e9ba688f56ab8f9"} Jul 08 20:11:14.780562 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:14.780361 2572 scope.go:117] "RemoveContainer" containerID="55720b7f55fc3a73d2b50c994cbf79eba475dff1f153bd2a97ca58fb89b71c93" Jul 08 20:11:14.817803 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:14.817778 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5"] Jul 08 20:11:14.823445 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:14.823425 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e1c198eb-d7b6c7d98-jvpx5"] Jul 08 20:11:15.673490 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:11:15.673458 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52fc569c-559e-44d1-88d8-6789b4302ba6" path="/var/lib/kubelet/pods/52fc569c-559e-44d1-88d8-6789b4302ba6/volumes" Jul 08 20:12:09.616394 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:09.616363 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:12:09.622093 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:09.622061 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:12:42.093868 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:42.093834 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:12:50.901535 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:50.901495 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:12:58.895780 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.895746 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb"] Jul 08 20:12:58.896123 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.896083 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52fc569c-559e-44d1-88d8-6789b4302ba6" containerName="maas-api" Jul 08 20:12:58.896123 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.896094 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="52fc569c-559e-44d1-88d8-6789b4302ba6" containerName="maas-api" Jul 08 20:12:58.896123 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.896103 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52fc569c-559e-44d1-88d8-6789b4302ba6" containerName="maas-api" Jul 08 20:12:58.896123 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.896112 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="52fc569c-559e-44d1-88d8-6789b4302ba6" containerName="maas-api" Jul 08 20:12:58.896275 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.896175 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="52fc569c-559e-44d1-88d8-6789b4302ba6" containerName="maas-api" Jul 08 20:12:58.896275 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.896190 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d8bbfc-876f-43dc-9921-7318fef899c7" containerName="maas-api" Jul 08 20:12:58.899077 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.899061 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:58.901308 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.901262 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:12:58.901441 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.901306 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-89ad9d7c-serving-cert\"" Jul 08 20:12:58.902048 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.902034 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:12:58.902108 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.902087 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:12:58.910019 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:58.909998 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb"] Jul 08 20:12:59.066989 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.066960 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1fba89-ae81-4295-bd4b-e264df735026-openshift-service-ca\") pod \"maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:59.067187 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.066999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1fba89-ae81-4295-bd4b-e264df735026-maas-api-tls\") pod \"maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:59.067187 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.067096 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4l6s\" (UniqueName: \"kubernetes.io/projected/8d1fba89-ae81-4295-bd4b-e264df735026-kube-api-access-d4l6s\") pod \"maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:59.168141 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.168070 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4l6s\" (UniqueName: \"kubernetes.io/projected/8d1fba89-ae81-4295-bd4b-e264df735026-kube-api-access-d4l6s\") pod \"maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:59.168141 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.168109 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1fba89-ae81-4295-bd4b-e264df735026-openshift-service-ca\") pod \"maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:59.168141 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.168133 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1fba89-ae81-4295-bd4b-e264df735026-maas-api-tls\") pod \"maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:59.168757 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.168732 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1fba89-ae81-4295-bd4b-e264df735026-openshift-service-ca\") pod \"maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:59.170503 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.170486 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1fba89-ae81-4295-bd4b-e264df735026-maas-api-tls\") pod \"maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:59.176009 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.175989 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4l6s\" (UniqueName: \"kubernetes.io/projected/8d1fba89-ae81-4295-bd4b-e264df735026-kube-api-access-d4l6s\") pod \"maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:59.210694 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.210667 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:12:59.335141 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:12:59.335057 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb"] Jul 08 20:12:59.337508 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:12:59.337475 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1fba89_ae81_4295_bd4b_e264df735026.slice/crio-7a4c4d8f153d63befeda2c09a9556489274e486a3773f7dcfe2502502b9443a1 WatchSource:0}: Error finding container 7a4c4d8f153d63befeda2c09a9556489274e486a3773f7dcfe2502502b9443a1: Status 404 returned error can't find the container with id 7a4c4d8f153d63befeda2c09a9556489274e486a3773f7dcfe2502502b9443a1 Jul 08 20:13:00.284695 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:00.284664 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" event={"ID":"8d1fba89-ae81-4295-bd4b-e264df735026","Type":"ContainerStarted","Data":"fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4"} Jul 08 20:13:00.285063 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:00.284715 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" event={"ID":"8d1fba89-ae81-4295-bd4b-e264df735026","Type":"ContainerStarted","Data":"7a4c4d8f153d63befeda2c09a9556489274e486a3773f7dcfe2502502b9443a1"} Jul 08 20:13:00.285063 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:00.284762 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:13:00.302647 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:00.302572 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" podStartSLOduration=1.596485682 podStartE2EDuration="2.30255811s" totalImagesPullingTime="706.072428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:13:00.299964807 +0000 UTC m=+1851.464515210" watchObservedRunningTime="2026-07-08 20:13:00.30255811 +0000 UTC m=+1851.467108511" Jul 08 20:13:04.787625 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:04.787593 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:13:06.295558 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:06.295526 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:13:10.828506 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:10.828471 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb"] Jul 08 20:13:10.828944 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:10.828727 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" podUID="8d1fba89-ae81-4295-bd4b-e264df735026" containerName="maas-api" containerID="cri-o://fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4" gracePeriod=30 Jul 08 20:13:11.067975 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.067952 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:13:11.266498 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.266389 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1fba89-ae81-4295-bd4b-e264df735026-openshift-service-ca\") pod \"8d1fba89-ae81-4295-bd4b-e264df735026\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " Jul 08 20:13:11.266498 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.266478 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1fba89-ae81-4295-bd4b-e264df735026-maas-api-tls\") pod \"8d1fba89-ae81-4295-bd4b-e264df735026\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " Jul 08 20:13:11.266729 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.266535 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4l6s\" (UniqueName: \"kubernetes.io/projected/8d1fba89-ae81-4295-bd4b-e264df735026-kube-api-access-d4l6s\") pod \"8d1fba89-ae81-4295-bd4b-e264df735026\" (UID: \"8d1fba89-ae81-4295-bd4b-e264df735026\") " Jul 08 20:13:11.266787 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.266764 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1fba89-ae81-4295-bd4b-e264df735026-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d1fba89-ae81-4295-bd4b-e264df735026" (UID: "8d1fba89-ae81-4295-bd4b-e264df735026"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:13:11.268553 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.268528 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1fba89-ae81-4295-bd4b-e264df735026-kube-api-access-d4l6s" (OuterVolumeSpecName: "kube-api-access-d4l6s") pod "8d1fba89-ae81-4295-bd4b-e264df735026" (UID: "8d1fba89-ae81-4295-bd4b-e264df735026"). InnerVolumeSpecName "kube-api-access-d4l6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:13:11.268677 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.268558 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1fba89-ae81-4295-bd4b-e264df735026-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d1fba89-ae81-4295-bd4b-e264df735026" (UID: "8d1fba89-ae81-4295-bd4b-e264df735026"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:13:11.339452 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.339418 2572 generic.go:356] "Generic (PLEG): container finished" podID="8d1fba89-ae81-4295-bd4b-e264df735026" containerID="fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4" exitCode=0 Jul 08 20:13:11.339592 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.339466 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" Jul 08 20:13:11.339592 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.339506 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" event={"ID":"8d1fba89-ae81-4295-bd4b-e264df735026","Type":"ContainerDied","Data":"fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4"} Jul 08 20:13:11.339592 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.339564 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb" event={"ID":"8d1fba89-ae81-4295-bd4b-e264df735026","Type":"ContainerDied","Data":"7a4c4d8f153d63befeda2c09a9556489274e486a3773f7dcfe2502502b9443a1"} Jul 08 20:13:11.339592 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.339587 2572 scope.go:117] "RemoveContainer" containerID="fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4" Jul 08 20:13:11.350954 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.350935 2572 scope.go:117] "RemoveContainer" containerID="fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4" Jul 08 20:13:11.351256 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:13:11.351238 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4\": container with ID starting with fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4 not found: ID does not exist" containerID="fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4" Jul 08 20:13:11.351335 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.351267 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4"} err="failed to get container status \"fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4\": rpc error: code = NotFound desc = could not find container \"fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4\": container with ID starting with fe89ff7166efc55309c86f4c9716c0ff807b940c4860ed667b55b11537d986a4 not found: ID does not exist" Jul 08 20:13:11.360261 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.360238 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb"] Jul 08 20:13:11.365170 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.365140 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-89ad9d7c-64f7b86bb7-b5kpb"] Jul 08 20:13:11.367499 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.367483 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1fba89-ae81-4295-bd4b-e264df735026-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:13:11.367555 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.367502 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d1fba89-ae81-4295-bd4b-e264df735026-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:13:11.367555 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.367511 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4l6s\" (UniqueName: \"kubernetes.io/projected/8d1fba89-ae81-4295-bd4b-e264df735026-kube-api-access-d4l6s\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:13:11.673365 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:11.673334 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d1fba89-ae81-4295-bd4b-e264df735026" path="/var/lib/kubelet/pods/8d1fba89-ae81-4295-bd4b-e264df735026/volumes" Jul 08 20:13:15.702038 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:15.702000 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:13:40.296644 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.296544 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2"] Jul 08 20:13:40.297426 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.297037 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1fba89-ae81-4295-bd4b-e264df735026" containerName="maas-api" Jul 08 20:13:40.297426 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.297053 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1fba89-ae81-4295-bd4b-e264df735026" containerName="maas-api" Jul 08 20:13:40.297426 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.297148 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="52fc569c-559e-44d1-88d8-6789b4302ba6" containerName="maas-api" Jul 08 20:13:40.297426 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.297176 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1fba89-ae81-4295-bd4b-e264df735026" containerName="maas-api" Jul 08 20:13:40.301654 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.301634 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.303854 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.303828 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-28e49818-serving-cert\"" Jul 08 20:13:40.303974 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.303888 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:13:40.303974 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.303837 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:13:40.304613 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.304597 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:13:40.309300 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.309275 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2"] Jul 08 20:13:40.397708 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.397671 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8br\" (UniqueName: \"kubernetes.io/projected/a149c885-423f-4ded-abb7-10b07e1cab68-kube-api-access-6p8br\") pod \"maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.397708 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.397710 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a149c885-423f-4ded-abb7-10b07e1cab68-maas-api-tls\") pod \"maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.397913 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.397748 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a149c885-423f-4ded-abb7-10b07e1cab68-openshift-service-ca\") pod \"maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.498961 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.498930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8br\" (UniqueName: \"kubernetes.io/projected/a149c885-423f-4ded-abb7-10b07e1cab68-kube-api-access-6p8br\") pod \"maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.498961 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.498968 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a149c885-423f-4ded-abb7-10b07e1cab68-maas-api-tls\") pod \"maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.499213 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.498995 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a149c885-423f-4ded-abb7-10b07e1cab68-openshift-service-ca\") pod \"maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.499627 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.499609 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a149c885-423f-4ded-abb7-10b07e1cab68-openshift-service-ca\") pod \"maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.501420 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.501402 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a149c885-423f-4ded-abb7-10b07e1cab68-maas-api-tls\") pod \"maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.508889 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.508865 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8br\" (UniqueName: \"kubernetes.io/projected/a149c885-423f-4ded-abb7-10b07e1cab68-kube-api-access-6p8br\") pod \"maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.613090 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.613000 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:40.754052 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:40.754014 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2"] Jul 08 20:13:40.758019 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:13:40.757993 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda149c885_423f_4ded_abb7_10b07e1cab68.slice/crio-d90aabecc91f14ed20eb500c608cef72b75250a0d0d729af5e2e850e028da1c7 WatchSource:0}: Error finding container d90aabecc91f14ed20eb500c608cef72b75250a0d0d729af5e2e850e028da1c7: Status 404 returned error can't find the container with id d90aabecc91f14ed20eb500c608cef72b75250a0d0d729af5e2e850e028da1c7 Jul 08 20:13:41.486823 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:41.486773 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" event={"ID":"a149c885-423f-4ded-abb7-10b07e1cab68","Type":"ContainerStarted","Data":"f4754ff177b3084aa5fe578d8f20dbb0d3980d226f62946f76e21460b3d232f2"} Jul 08 20:13:41.487223 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:41.486957 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:13:41.487223 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:41.486984 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" event={"ID":"a149c885-423f-4ded-abb7-10b07e1cab68","Type":"ContainerStarted","Data":"d90aabecc91f14ed20eb500c608cef72b75250a0d0d729af5e2e850e028da1c7"} Jul 08 20:13:41.503747 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:41.503693 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" podStartSLOduration=0.998149054 podStartE2EDuration="1.503677564s" totalImagesPullingTime="505.52851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:13:41.502537026 +0000 UTC m=+1892.667087429" watchObservedRunningTime="2026-07-08 20:13:41.503677564 +0000 UTC m=+1892.668227970" Jul 08 20:13:46.587828 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:46.587789 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:13:47.497848 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:13:47.497815 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:14:02.082762 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.082726 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj"] Jul 08 20:14:02.086085 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.086068 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.088336 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.088316 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-28e49818\"/\"kube-root-ca.crt\"" Jul 08 20:14:02.088464 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.088378 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-28e49818\"/\"openshift-service-ca.crt\"" Jul 08 20:14:02.089201 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.089121 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-28e49818\"/\"default-dockercfg-bbtjc\"" Jul 08 20:14:02.089201 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.089183 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-28e49818\"/\"e2e-lifecycle-model-28e49818-kserve-self-signed-certs\"" Jul 08 20:14:02.095529 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.095505 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj"] Jul 08 20:14:02.177893 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.177860 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-dshm\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.178009 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.177909 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-tmp-dir\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.178009 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.177939 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-model-cache\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.178009 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.177985 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v996k\" (UniqueName: \"kubernetes.io/projected/66213c1d-c168-4bba-9e3b-cfff40d7c086-kube-api-access-v996k\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.178184 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.178016 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-home\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.178184 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.178044 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66213c1d-c168-4bba-9e3b-cfff40d7c086-tls-certs\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.278416 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.278388 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v996k\" (UniqueName: \"kubernetes.io/projected/66213c1d-c168-4bba-9e3b-cfff40d7c086-kube-api-access-v996k\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.278521 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.278427 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-home\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.278521 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.278467 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66213c1d-c168-4bba-9e3b-cfff40d7c086-tls-certs\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.278623 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.278521 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-dshm\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.278623 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.278542 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-tmp-dir\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.278623 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.278567 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-model-cache\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.278867 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.278843 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-home\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.278929 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.278868 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-tmp-dir\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.278969 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.278925 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-model-cache\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.280811 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.280791 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-dshm\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.280911 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.280894 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66213c1d-c168-4bba-9e3b-cfff40d7c086-tls-certs\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.285971 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.285951 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v996k\" (UniqueName: \"kubernetes.io/projected/66213c1d-c168-4bba-9e3b-cfff40d7c086-kube-api-access-v996k\") pod \"e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.397385 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.397325 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:02.528700 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.528673 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj"] Jul 08 20:14:02.531412 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:14:02.531386 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66213c1d_c168_4bba_9e3b_cfff40d7c086.slice/crio-5e8ac6573704e5f7cd105a57c914e23c0b01ba68b672795ff7810cf4ebb6ec10 WatchSource:0}: Error finding container 5e8ac6573704e5f7cd105a57c914e23c0b01ba68b672795ff7810cf4ebb6ec10: Status 404 returned error can't find the container with id 5e8ac6573704e5f7cd105a57c914e23c0b01ba68b672795ff7810cf4ebb6ec10 Jul 08 20:14:02.587145 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:02.587121 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" event={"ID":"66213c1d-c168-4bba-9e3b-cfff40d7c086","Type":"ContainerStarted","Data":"5e8ac6573704e5f7cd105a57c914e23c0b01ba68b672795ff7810cf4ebb6ec10"} Jul 08 20:14:03.593976 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:03.593937 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" event={"ID":"66213c1d-c168-4bba-9e3b-cfff40d7c086","Type":"ContainerStarted","Data":"77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175"} Jul 08 20:14:03.609768 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:03.609723 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" podStartSLOduration=1.609708417 podStartE2EDuration="1.609708417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:14:03.608500384 +0000 UTC m=+1914.773050789" watchObservedRunningTime="2026-07-08 20:14:03.609708417 +0000 UTC m=+1914.774258820" Jul 08 20:14:05.493531 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:05.493496 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:14:12.398006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:12.397972 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:12.398421 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:12.398044 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:12.410434 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:12.410412 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:12.654466 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:12.654381 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:19.785275 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:19.785243 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:14:22.486387 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.486350 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2"] Jul 08 20:14:22.486841 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.486630 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" podUID="a149c885-423f-4ded-abb7-10b07e1cab68" containerName="maas-api" containerID="cri-o://f4754ff177b3084aa5fe578d8f20dbb0d3980d226f62946f76e21460b3d232f2" gracePeriod=30 Jul 08 20:14:22.492651 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.492617 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" podUID="a149c885-423f-4ded-abb7-10b07e1cab68" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.43:8443/health\": dial tcp 10.134.0.43:8443: connect: connection refused" Jul 08 20:14:22.693062 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.693032 2572 generic.go:356] "Generic (PLEG): container finished" podID="a149c885-423f-4ded-abb7-10b07e1cab68" containerID="f4754ff177b3084aa5fe578d8f20dbb0d3980d226f62946f76e21460b3d232f2" exitCode=0 Jul 08 20:14:22.693258 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.693097 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" event={"ID":"a149c885-423f-4ded-abb7-10b07e1cab68","Type":"ContainerDied","Data":"f4754ff177b3084aa5fe578d8f20dbb0d3980d226f62946f76e21460b3d232f2"} Jul 08 20:14:22.731418 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.731400 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:14:22.732376 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.732359 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a149c885-423f-4ded-abb7-10b07e1cab68-openshift-service-ca\") pod \"a149c885-423f-4ded-abb7-10b07e1cab68\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " Jul 08 20:14:22.732456 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.732445 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p8br\" (UniqueName: \"kubernetes.io/projected/a149c885-423f-4ded-abb7-10b07e1cab68-kube-api-access-6p8br\") pod \"a149c885-423f-4ded-abb7-10b07e1cab68\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " Jul 08 20:14:22.732502 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.732470 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a149c885-423f-4ded-abb7-10b07e1cab68-maas-api-tls\") pod \"a149c885-423f-4ded-abb7-10b07e1cab68\" (UID: \"a149c885-423f-4ded-abb7-10b07e1cab68\") " Jul 08 20:14:22.732768 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.732733 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a149c885-423f-4ded-abb7-10b07e1cab68-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a149c885-423f-4ded-abb7-10b07e1cab68" (UID: "a149c885-423f-4ded-abb7-10b07e1cab68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:14:22.734422 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.734400 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a149c885-423f-4ded-abb7-10b07e1cab68-kube-api-access-6p8br" (OuterVolumeSpecName: "kube-api-access-6p8br") pod "a149c885-423f-4ded-abb7-10b07e1cab68" (UID: "a149c885-423f-4ded-abb7-10b07e1cab68"). InnerVolumeSpecName "kube-api-access-6p8br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:14:22.734514 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.734435 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a149c885-423f-4ded-abb7-10b07e1cab68-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a149c885-423f-4ded-abb7-10b07e1cab68" (UID: "a149c885-423f-4ded-abb7-10b07e1cab68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:14:22.833028 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.832997 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a149c885-423f-4ded-abb7-10b07e1cab68-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:14:22.833028 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.833026 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p8br\" (UniqueName: \"kubernetes.io/projected/a149c885-423f-4ded-abb7-10b07e1cab68-kube-api-access-6p8br\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:14:22.833235 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:22.833042 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a149c885-423f-4ded-abb7-10b07e1cab68-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:14:23.698316 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:23.698288 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" event={"ID":"a149c885-423f-4ded-abb7-10b07e1cab68","Type":"ContainerDied","Data":"d90aabecc91f14ed20eb500c608cef72b75250a0d0d729af5e2e850e028da1c7"} Jul 08 20:14:23.698681 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:23.698318 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2" Jul 08 20:14:23.698681 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:23.698353 2572 scope.go:117] "RemoveContainer" containerID="f4754ff177b3084aa5fe578d8f20dbb0d3980d226f62946f76e21460b3d232f2" Jul 08 20:14:23.713352 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:23.713330 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2"] Jul 08 20:14:23.717282 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:23.717253 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-28e49818-7cd5b4bb77-mjxs2"] Jul 08 20:14:25.673364 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:25.673332 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a149c885-423f-4ded-abb7-10b07e1cab68" path="/var/lib/kubelet/pods/a149c885-423f-4ded-abb7-10b07e1cab68/volumes" Jul 08 20:14:32.693062 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:32.693032 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:14:33.133433 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:33.133401 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj"] Jul 08 20:14:48.154054 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.154014 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" podUID="66213c1d-c168-4bba-9e3b-cfff40d7c086" containerName="main" containerID="cri-o://77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175" gracePeriod=45 Jul 08 20:14:48.402712 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.402688 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:48.438008 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.437943 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-dshm\") pod \"66213c1d-c168-4bba-9e3b-cfff40d7c086\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " Jul 08 20:14:48.438008 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.437974 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-model-cache\") pod \"66213c1d-c168-4bba-9e3b-cfff40d7c086\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " Jul 08 20:14:48.438008 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.437995 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-home\") pod \"66213c1d-c168-4bba-9e3b-cfff40d7c086\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " Jul 08 20:14:48.438312 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.438057 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v996k\" (UniqueName: \"kubernetes.io/projected/66213c1d-c168-4bba-9e3b-cfff40d7c086-kube-api-access-v996k\") pod \"66213c1d-c168-4bba-9e3b-cfff40d7c086\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " Jul 08 20:14:48.438312 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.438141 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66213c1d-c168-4bba-9e3b-cfff40d7c086-tls-certs\") pod \"66213c1d-c168-4bba-9e3b-cfff40d7c086\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " Jul 08 20:14:48.438312 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.438199 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-tmp-dir\") pod \"66213c1d-c168-4bba-9e3b-cfff40d7c086\" (UID: \"66213c1d-c168-4bba-9e3b-cfff40d7c086\") " Jul 08 20:14:48.438312 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.438253 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-model-cache" (OuterVolumeSpecName: "model-cache") pod "66213c1d-c168-4bba-9e3b-cfff40d7c086" (UID: "66213c1d-c168-4bba-9e3b-cfff40d7c086"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:14:48.438548 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.438314 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-home" (OuterVolumeSpecName: "home") pod "66213c1d-c168-4bba-9e3b-cfff40d7c086" (UID: "66213c1d-c168-4bba-9e3b-cfff40d7c086"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:14:48.438548 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.438472 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-model-cache\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:14:48.438548 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.438489 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-home\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:14:48.438548 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.438509 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66213c1d-c168-4bba-9e3b-cfff40d7c086" (UID: "66213c1d-c168-4bba-9e3b-cfff40d7c086"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:14:48.440110 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.440086 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-dshm" (OuterVolumeSpecName: "dshm") pod "66213c1d-c168-4bba-9e3b-cfff40d7c086" (UID: "66213c1d-c168-4bba-9e3b-cfff40d7c086"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:14:48.440628 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.440598 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66213c1d-c168-4bba-9e3b-cfff40d7c086-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66213c1d-c168-4bba-9e3b-cfff40d7c086" (UID: "66213c1d-c168-4bba-9e3b-cfff40d7c086"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:14:48.440628 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.440604 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66213c1d-c168-4bba-9e3b-cfff40d7c086-kube-api-access-v996k" (OuterVolumeSpecName: "kube-api-access-v996k") pod "66213c1d-c168-4bba-9e3b-cfff40d7c086" (UID: "66213c1d-c168-4bba-9e3b-cfff40d7c086"). InnerVolumeSpecName "kube-api-access-v996k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:14:48.538941 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.538907 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66213c1d-c168-4bba-9e3b-cfff40d7c086-tls-certs\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:14:48.538941 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.538949 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-tmp-dir\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:14:48.539149 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.538959 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66213c1d-c168-4bba-9e3b-cfff40d7c086-dshm\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:14:48.539149 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.538967 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v996k\" (UniqueName: \"kubernetes.io/projected/66213c1d-c168-4bba-9e3b-cfff40d7c086-kube-api-access-v996k\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:14:48.823121 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.823082 2572 generic.go:356] "Generic (PLEG): container finished" podID="66213c1d-c168-4bba-9e3b-cfff40d7c086" containerID="77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175" exitCode=0 Jul 08 20:14:48.823304 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.823125 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" Jul 08 20:14:48.823304 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.823189 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" event={"ID":"66213c1d-c168-4bba-9e3b-cfff40d7c086","Type":"ContainerDied","Data":"77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175"} Jul 08 20:14:48.823304 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.823230 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj" event={"ID":"66213c1d-c168-4bba-9e3b-cfff40d7c086","Type":"ContainerDied","Data":"5e8ac6573704e5f7cd105a57c914e23c0b01ba68b672795ff7810cf4ebb6ec10"} Jul 08 20:14:48.823304 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.823248 2572 scope.go:117] "RemoveContainer" containerID="77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175" Jul 08 20:14:48.838016 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.837997 2572 scope.go:117] "RemoveContainer" containerID="77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175" Jul 08 20:14:48.838360 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:14:48.838321 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175\": container with ID starting with 77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175 not found: ID does not exist" containerID="77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175" Jul 08 20:14:48.838434 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.838382 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175"} err="failed to get container status \"77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175\": rpc error: code = NotFound desc = could not find container \"77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175\": container with ID starting with 77c03a05a4537e9266b14c0598e59c2b102615a8e6c37e8e8c7f8224ca76a175 not found: ID does not exist" Jul 08 20:14:48.847709 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.847689 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj"] Jul 08 20:14:48.851139 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:48.851119 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-28e49818/e2e-lifecycle-model-28e49818-kserve-77bd8fb69f-vxzwj"] Jul 08 20:14:49.673225 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:14:49.673190 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66213c1d-c168-4bba-9e3b-cfff40d7c086" path="/var/lib/kubelet/pods/66213c1d-c168-4bba-9e3b-cfff40d7c086/volumes" Jul 08 20:15:00.137631 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.137602 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w"] Jul 08 20:15:00.138068 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.137893 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a149c885-423f-4ded-abb7-10b07e1cab68" containerName="maas-api" Jul 08 20:15:00.138068 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.137905 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a149c885-423f-4ded-abb7-10b07e1cab68" containerName="maas-api" Jul 08 20:15:00.138068 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.137917 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66213c1d-c168-4bba-9e3b-cfff40d7c086" containerName="main" Jul 08 20:15:00.138068 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.137923 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="66213c1d-c168-4bba-9e3b-cfff40d7c086" containerName="main" Jul 08 20:15:00.138068 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.137982 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="66213c1d-c168-4bba-9e3b-cfff40d7c086" containerName="main" Jul 08 20:15:00.138068 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.137989 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a149c885-423f-4ded-abb7-10b07e1cab68" containerName="maas-api" Jul 08 20:15:00.142029 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.142014 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" Jul 08 20:15:00.144720 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.144703 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:15:00.144779 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.144703 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:15:00.144824 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.144780 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:15:00.150352 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.150331 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w"] Jul 08 20:15:00.230402 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.230377 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bv89\" (UniqueName: \"kubernetes.io/projected/99bf3d3c-241a-4d4c-acb2-fa50b7800f93-kube-api-access-2bv89\") pod \"maas-api-key-cleanup-29725695-9h28w\" (UID: \"99bf3d3c-241a-4d4c-acb2-fa50b7800f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" Jul 08 20:15:00.331176 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.331111 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bv89\" (UniqueName: \"kubernetes.io/projected/99bf3d3c-241a-4d4c-acb2-fa50b7800f93-kube-api-access-2bv89\") pod \"maas-api-key-cleanup-29725695-9h28w\" (UID: \"99bf3d3c-241a-4d4c-acb2-fa50b7800f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" Jul 08 20:15:00.339364 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.339336 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bv89\" (UniqueName: \"kubernetes.io/projected/99bf3d3c-241a-4d4c-acb2-fa50b7800f93-kube-api-access-2bv89\") pod \"maas-api-key-cleanup-29725695-9h28w\" (UID: \"99bf3d3c-241a-4d4c-acb2-fa50b7800f93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" Jul 08 20:15:00.452586 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.452537 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" Jul 08 20:15:00.574198 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.574171 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w"] Jul 08 20:15:00.575735 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:15:00.575707 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bf3d3c_241a_4d4c_acb2_fa50b7800f93.slice/crio-7f6c36b72412518032fa8b405a03ffa19ca4236bcb39253abd2cbe4ec352f213 WatchSource:0}: Error finding container 7f6c36b72412518032fa8b405a03ffa19ca4236bcb39253abd2cbe4ec352f213: Status 404 returned error can't find the container with id 7f6c36b72412518032fa8b405a03ffa19ca4236bcb39253abd2cbe4ec352f213 Jul 08 20:15:00.881758 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.881727 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" event={"ID":"99bf3d3c-241a-4d4c-acb2-fa50b7800f93","Type":"ContainerStarted","Data":"dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543"} Jul 08 20:15:00.881925 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.881776 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" event={"ID":"99bf3d3c-241a-4d4c-acb2-fa50b7800f93","Type":"ContainerStarted","Data":"7f6c36b72412518032fa8b405a03ffa19ca4236bcb39253abd2cbe4ec352f213"} Jul 08 20:15:00.895701 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:00.895650 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" podStartSLOduration=0.895632834 podStartE2EDuration="895.632834ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:15:00.894677386 +0000 UTC m=+1972.059227799" watchObservedRunningTime="2026-07-08 20:15:00.895632834 +0000 UTC m=+1972.060183300" Jul 08 20:15:13.188822 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:13.188782 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:15:43.401041 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:43.401002 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:15:55.409809 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:15:55.409776 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:16:00.011027 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:00.010978 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w"] Jul 08 20:16:00.011570 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:00.011245 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" podUID="99bf3d3c-241a-4d4c-acb2-fa50b7800f93" containerName="cleanup" containerID="cri-o://dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543" gracePeriod=30 Jul 08 20:16:30.186616 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.186590 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" Jul 08 20:16:30.259702 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.259669 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bv89\" (UniqueName: \"kubernetes.io/projected/99bf3d3c-241a-4d4c-acb2-fa50b7800f93-kube-api-access-2bv89\") pod \"99bf3d3c-241a-4d4c-acb2-fa50b7800f93\" (UID: \"99bf3d3c-241a-4d4c-acb2-fa50b7800f93\") " Jul 08 20:16:30.261642 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.261614 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bf3d3c-241a-4d4c-acb2-fa50b7800f93-kube-api-access-2bv89" (OuterVolumeSpecName: "kube-api-access-2bv89") pod "99bf3d3c-241a-4d4c-acb2-fa50b7800f93" (UID: "99bf3d3c-241a-4d4c-acb2-fa50b7800f93"). InnerVolumeSpecName "kube-api-access-2bv89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:16:30.320298 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.320234 2572 generic.go:356] "Generic (PLEG): container finished" podID="99bf3d3c-241a-4d4c-acb2-fa50b7800f93" containerID="dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543" exitCode=137 Jul 08 20:16:30.320298 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.320282 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" Jul 08 20:16:30.320461 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.320317 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" event={"ID":"99bf3d3c-241a-4d4c-acb2-fa50b7800f93","Type":"ContainerDied","Data":"dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543"} Jul 08 20:16:30.320461 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.320371 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w" event={"ID":"99bf3d3c-241a-4d4c-acb2-fa50b7800f93","Type":"ContainerDied","Data":"7f6c36b72412518032fa8b405a03ffa19ca4236bcb39253abd2cbe4ec352f213"} Jul 08 20:16:30.320461 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.320398 2572 scope.go:117] "RemoveContainer" containerID="dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543" Jul 08 20:16:30.329472 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.329457 2572 scope.go:117] "RemoveContainer" containerID="dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543" Jul 08 20:16:30.329706 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:16:30.329685 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543\": container with ID starting with dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543 not found: ID does not exist" containerID="dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543" Jul 08 20:16:30.329755 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.329715 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543"} err="failed to get container status \"dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543\": rpc error: code = NotFound desc = could not find container \"dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543\": container with ID starting with dda2e759547018cb0cb3ebe44e57dec95cbcf4c154344242aee3384c6973d543 not found: ID does not exist" Jul 08 20:16:30.340534 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.340510 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w"] Jul 08 20:16:30.344004 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.343986 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725695-9h28w"] Jul 08 20:16:30.360322 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:30.360298 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bv89\" (UniqueName: \"kubernetes.io/projected/99bf3d3c-241a-4d4c-acb2-fa50b7800f93-kube-api-access-2bv89\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:16:31.673486 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:31.673453 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99bf3d3c-241a-4d4c-acb2-fa50b7800f93" path="/var/lib/kubelet/pods/99bf3d3c-241a-4d4c-acb2-fa50b7800f93/volumes" Jul 08 20:16:37.025862 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.025824 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj"] Jul 08 20:16:37.026361 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.026280 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99bf3d3c-241a-4d4c-acb2-fa50b7800f93" containerName="cleanup" Jul 08 20:16:37.026361 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.026295 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="99bf3d3c-241a-4d4c-acb2-fa50b7800f93" containerName="cleanup" Jul 08 20:16:37.026490 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.026393 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="99bf3d3c-241a-4d4c-acb2-fa50b7800f93" containerName="cleanup" Jul 08 20:16:37.031194 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.031171 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.034911 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.034891 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-db100e-serving-cert\"" Jul 08 20:16:37.035034 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.034910 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:16:37.035455 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.035435 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:16:37.035560 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.035539 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:16:37.049612 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.049593 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj"] Jul 08 20:16:37.120873 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.120834 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rnng\" (UniqueName: \"kubernetes.io/projected/d6006db9-095c-4670-b632-58c6f8127e1c-kube-api-access-5rnng\") pod \"maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.121075 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.120904 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6006db9-095c-4670-b632-58c6f8127e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.121075 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.120993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6006db9-095c-4670-b632-58c6f8127e1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.221702 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.221669 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6006db9-095c-4670-b632-58c6f8127e1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.221873 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.221712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rnng\" (UniqueName: \"kubernetes.io/projected/d6006db9-095c-4670-b632-58c6f8127e1c-kube-api-access-5rnng\") pod \"maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.221873 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.221750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6006db9-095c-4670-b632-58c6f8127e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.221873 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:16:37.221856 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-serving-cert: secret "maas-api-e2e-inf-a-db100e-serving-cert" not found Jul 08 20:16:37.222049 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:16:37.221913 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6006db9-095c-4670-b632-58c6f8127e1c-maas-api-tls podName:d6006db9-095c-4670-b632-58c6f8127e1c nodeName:}" failed. No retries permitted until 2026-07-08 20:16:37.721896483 +0000 UTC m=+2068.886446868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6006db9-095c-4670-b632-58c6f8127e1c-maas-api-tls") pod "maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" (UID: "d6006db9-095c-4670-b632-58c6f8127e1c") : secret "maas-api-e2e-inf-a-db100e-serving-cert" not found Jul 08 20:16:37.222356 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.222334 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6006db9-095c-4670-b632-58c6f8127e1c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.231976 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.231950 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rnng\" (UniqueName: \"kubernetes.io/projected/d6006db9-095c-4670-b632-58c6f8127e1c-kube-api-access-5rnng\") pod \"maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.725991 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.725957 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6006db9-095c-4670-b632-58c6f8127e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.728362 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.728344 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6006db9-095c-4670-b632-58c6f8127e1c-maas-api-tls\") pod \"maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:37.941483 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:37.941438 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:38.069512 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:38.069478 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj"] Jul 08 20:16:38.073240 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:16:38.073215 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6006db9_095c_4670_b632_58c6f8127e1c.slice/crio-b51f31522422cef0deb6cafe8529d3b98c1fc7b290c6b14449662930ed5acf2e WatchSource:0}: Error finding container b51f31522422cef0deb6cafe8529d3b98c1fc7b290c6b14449662930ed5acf2e: Status 404 returned error can't find the container with id b51f31522422cef0deb6cafe8529d3b98c1fc7b290c6b14449662930ed5acf2e Jul 08 20:16:38.074537 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:38.074520 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 20:16:38.360971 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:38.360939 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" event={"ID":"d6006db9-095c-4670-b632-58c6f8127e1c","Type":"ContainerStarted","Data":"b51f31522422cef0deb6cafe8529d3b98c1fc7b290c6b14449662930ed5acf2e"} Jul 08 20:16:39.367732 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:39.367703 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" event={"ID":"d6006db9-095c-4670-b632-58c6f8127e1c","Type":"ContainerStarted","Data":"88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf"} Jul 08 20:16:39.368117 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:39.367839 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:39.383914 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:39.383873 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" podStartSLOduration=2.737032691 podStartE2EDuration="3.38385921s" totalImagesPullingTime="646.826519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:16:39.382638223 +0000 UTC m=+2070.547188717" watchObservedRunningTime="2026-07-08 20:16:39.38385921 +0000 UTC m=+2070.548409614" Jul 08 20:16:44.975594 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:44.975558 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m"] Jul 08 20:16:44.979372 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:44.979352 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:44.981501 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:44.981477 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4eb9fe-serving-cert\"" Jul 08 20:16:44.989212 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:44.989191 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m"] Jul 08 20:16:45.003435 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.003410 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:16:45.088512 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.088486 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck82l\" (UniqueName: \"kubernetes.io/projected/1744fb44-03eb-4276-8e68-e3ca279747b6-kube-api-access-ck82l\") pod \"maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:45.088632 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.088616 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1744fb44-03eb-4276-8e68-e3ca279747b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:45.088687 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.088667 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1744fb44-03eb-4276-8e68-e3ca279747b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:45.189273 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.189245 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1744fb44-03eb-4276-8e68-e3ca279747b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:45.189408 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.189286 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1744fb44-03eb-4276-8e68-e3ca279747b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:45.189408 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.189324 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck82l\" (UniqueName: \"kubernetes.io/projected/1744fb44-03eb-4276-8e68-e3ca279747b6-kube-api-access-ck82l\") pod \"maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:45.189531 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:16:45.189413 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-serving-cert: secret "maas-api-e2e-inf-b-4eb9fe-serving-cert" not found Jul 08 20:16:45.189531 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:16:45.189476 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1744fb44-03eb-4276-8e68-e3ca279747b6-maas-api-tls podName:1744fb44-03eb-4276-8e68-e3ca279747b6 nodeName:}" failed. No retries permitted until 2026-07-08 20:16:45.689459245 +0000 UTC m=+2076.854009633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1744fb44-03eb-4276-8e68-e3ca279747b6-maas-api-tls") pod "maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" (UID: "1744fb44-03eb-4276-8e68-e3ca279747b6") : secret "maas-api-e2e-inf-b-4eb9fe-serving-cert" not found Jul 08 20:16:45.189906 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.189886 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1744fb44-03eb-4276-8e68-e3ca279747b6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:45.197710 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.197692 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck82l\" (UniqueName: \"kubernetes.io/projected/1744fb44-03eb-4276-8e68-e3ca279747b6-kube-api-access-ck82l\") pod \"maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:45.377944 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.377920 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:16:45.693908 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.693821 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1744fb44-03eb-4276-8e68-e3ca279747b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:45.696335 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.696312 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1744fb44-03eb-4276-8e68-e3ca279747b6-maas-api-tls\") pod \"maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:45.892691 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:45.892649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:46.028325 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:46.028293 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m"] Jul 08 20:16:46.030462 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:16:46.030439 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1744fb44_03eb_4276_8e68_e3ca279747b6.slice/crio-b8dea17c9b8dfb61b0d25ccddb9d9ad8e55d918f33b902b23dd2cda9d80ce403 WatchSource:0}: Error finding container b8dea17c9b8dfb61b0d25ccddb9d9ad8e55d918f33b902b23dd2cda9d80ce403: Status 404 returned error can't find the container with id b8dea17c9b8dfb61b0d25ccddb9d9ad8e55d918f33b902b23dd2cda9d80ce403 Jul 08 20:16:46.404467 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:46.404433 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" event={"ID":"1744fb44-03eb-4276-8e68-e3ca279747b6","Type":"ContainerStarted","Data":"b8dea17c9b8dfb61b0d25ccddb9d9ad8e55d918f33b902b23dd2cda9d80ce403"} Jul 08 20:16:47.410815 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:47.410778 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" event={"ID":"1744fb44-03eb-4276-8e68-e3ca279747b6","Type":"ContainerStarted","Data":"43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda"} Jul 08 20:16:47.411273 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:47.410890 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:47.427846 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:47.427797 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" podStartSLOduration=3.07302128 podStartE2EDuration="3.42777916s" totalImagesPullingTime="354.75788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:16:47.42619788 +0000 UTC m=+2078.590748283" watchObservedRunningTime="2026-07-08 20:16:47.42777916 +0000 UTC m=+2078.592329562" Jul 08 20:16:53.421228 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:53.421198 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:16:54.135030 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.134996 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l"] Jul 08 20:16:54.138555 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.138534 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.142474 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.142450 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-db100e\"/\"kube-root-ca.crt\"" Jul 08 20:16:54.142720 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.142700 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-db100e\"/\"openshift-service-ca.crt\"" Jul 08 20:16:54.143518 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.143496 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-db100e\"/\"default-dockercfg-4zjwg\"" Jul 08 20:16:54.143996 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.143975 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-db100e\"/\"test-model-db100e-kserve-self-signed-certs\"" Jul 08 20:16:54.151944 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.151919 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l"] Jul 08 20:16:54.269554 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.269514 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-model-cache\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.269554 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.269559 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tmp-dir\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.269775 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.269586 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gflfp\" (UniqueName: \"kubernetes.io/projected/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-kube-api-access-gflfp\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.269775 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.269617 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-home\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.269775 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.269637 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tls-certs\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.269775 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.269664 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-dshm\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.370380 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.370340 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-model-cache\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.370380 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.370389 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tmp-dir\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.370617 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.370412 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gflfp\" (UniqueName: \"kubernetes.io/projected/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-kube-api-access-gflfp\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.370617 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.370512 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-home\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.370617 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.370564 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tls-certs\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.370617 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.370609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-dshm\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.370873 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.370843 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tmp-dir\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.370941 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.370861 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-model-cache\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.370941 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.370894 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-home\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.372874 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.372844 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-dshm\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.373150 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.373131 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tls-certs\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.377780 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.377759 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gflfp\" (UniqueName: \"kubernetes.io/projected/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-kube-api-access-gflfp\") pod \"test-model-db100e-kserve-69bff44f85-46x4l\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.449558 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.449466 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:16:54.582957 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:54.582930 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l"] Jul 08 20:16:54.584925 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:16:54.584892 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf08e93dc_796c_4378_8e8e_b8bb29a7e9c2.slice/crio-bc1a704b4783978a783070b3de337df6a80c5d98c3887752fc5d0ae7060b9680 WatchSource:0}: Error finding container bc1a704b4783978a783070b3de337df6a80c5d98c3887752fc5d0ae7060b9680: Status 404 returned error can't find the container with id bc1a704b4783978a783070b3de337df6a80c5d98c3887752fc5d0ae7060b9680 Jul 08 20:16:55.454509 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:55.454475 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" event={"ID":"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2","Type":"ContainerStarted","Data":"929dbfda79a8b82f8457b76721a5874854faa340e17294b749291afc013b57b8"} Jul 08 20:16:55.454509 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:55.454518 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" event={"ID":"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2","Type":"ContainerStarted","Data":"bc1a704b4783978a783070b3de337df6a80c5d98c3887752fc5d0ae7060b9680"} Jul 08 20:16:55.472732 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:16:55.472689 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" podStartSLOduration=1.472674462 podStartE2EDuration="1.472674462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:16:55.470989675 +0000 UTC m=+2086.635540080" watchObservedRunningTime="2026-07-08 20:16:55.472674462 +0000 UTC m=+2086.637224866" Jul 08 20:17:00.403673 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:00.403627 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:17:04.450338 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:04.450292 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:17:04.450826 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:04.450351 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:17:04.462530 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:04.462507 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:17:04.510676 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:04.510653 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:17:09.648531 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:09.648502 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:17:09.656136 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:09.656116 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:17:19.297461 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.297429 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9"] Jul 08 20:17:19.300641 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.300625 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.302757 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.302735 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4eb9fe\"/\"openshift-service-ca.crt\"" Jul 08 20:17:19.303532 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.303513 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4eb9fe\"/\"default-dockercfg-nwknz\"" Jul 08 20:17:19.303651 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.303572 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4eb9fe\"/\"kube-root-ca.crt\"" Jul 08 20:17:19.303721 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.303648 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4eb9fe\"/\"test-model-4eb9fe-kserve-self-signed-certs\"" Jul 08 20:17:19.311063 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.311042 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9"] Jul 08 20:17:19.389321 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.389294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-home\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.389459 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.389333 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4svr5\" (UniqueName: \"kubernetes.io/projected/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-kube-api-access-4svr5\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.389459 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.389360 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tls-certs\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.389459 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.389376 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tmp-dir\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.389459 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.389442 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-dshm\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.389600 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.389487 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-model-cache\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.490840 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.490812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-home\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.491004 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.490853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svr5\" (UniqueName: \"kubernetes.io/projected/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-kube-api-access-4svr5\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.491004 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.490874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tls-certs\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.491004 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.490907 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tmp-dir\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.491004 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.490930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-dshm\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.491004 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.490954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-model-cache\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.491283 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.491262 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-home\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.491329 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.491297 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tmp-dir\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.491369 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.491341 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-model-cache\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.493248 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.493229 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-dshm\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.493507 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.493488 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tls-certs\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.498555 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.498535 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svr5\" (UniqueName: \"kubernetes.io/projected/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-kube-api-access-4svr5\") pod \"test-model-4eb9fe-kserve-577c6589dc-4znz9\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.611078 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.610997 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:19.747040 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:19.747010 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9"] Jul 08 20:17:19.748398 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:17:19.748373 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1a1f911_6ac9_4d6a_97e7_46987634e1fa.slice/crio-bdeeed11460705ce52e001a1123c051146f49c6b45b66361c7c24c14215e11ad WatchSource:0}: Error finding container bdeeed11460705ce52e001a1123c051146f49c6b45b66361c7c24c14215e11ad: Status 404 returned error can't find the container with id bdeeed11460705ce52e001a1123c051146f49c6b45b66361c7c24c14215e11ad Jul 08 20:17:20.582807 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:20.582774 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" event={"ID":"f1a1f911-6ac9-4d6a-97e7-46987634e1fa","Type":"ContainerStarted","Data":"c7f37f878907384fdf4b5645b24dc7e871317eb51bda2a79ac29d98562c4963b"} Jul 08 20:17:20.583242 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:20.582817 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" event={"ID":"f1a1f911-6ac9-4d6a-97e7-46987634e1fa","Type":"ContainerStarted","Data":"bdeeed11460705ce52e001a1123c051146f49c6b45b66361c7c24c14215e11ad"} Jul 08 20:17:20.598972 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:20.598931 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" podStartSLOduration=1.598918898 podStartE2EDuration="1.598918898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:17:20.597588548 +0000 UTC m=+2111.762138962" watchObservedRunningTime="2026-07-08 20:17:20.598918898 +0000 UTC m=+2111.763469301" Jul 08 20:17:24.288070 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:24.288035 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:17:29.611176 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:29.611105 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:29.611176 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:29.611179 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:29.623470 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:29.623443 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:29.637607 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:29.637570 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:37.575670 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:37.575634 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l"] Jul 08 20:17:38.506836 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:38.506805 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9"] Jul 08 20:17:39.355214 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.355183 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj"] Jul 08 20:17:39.355677 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.355456 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" podUID="d6006db9-095c-4670-b632-58c6f8127e1c" containerName="maas-api" containerID="cri-o://88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf" gracePeriod=30 Jul 08 20:17:39.604875 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.604852 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:17:39.677461 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.677390 2572 generic.go:356] "Generic (PLEG): container finished" podID="d6006db9-095c-4670-b632-58c6f8127e1c" containerID="88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf" exitCode=0 Jul 08 20:17:39.677461 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.677416 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" Jul 08 20:17:39.677660 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.677458 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" event={"ID":"d6006db9-095c-4670-b632-58c6f8127e1c","Type":"ContainerDied","Data":"88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf"} Jul 08 20:17:39.677660 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.677510 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj" event={"ID":"d6006db9-095c-4670-b632-58c6f8127e1c","Type":"ContainerDied","Data":"b51f31522422cef0deb6cafe8529d3b98c1fc7b290c6b14449662930ed5acf2e"} Jul 08 20:17:39.677660 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.677538 2572 scope.go:117] "RemoveContainer" containerID="88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf" Jul 08 20:17:39.685961 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.685947 2572 scope.go:117] "RemoveContainer" containerID="88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf" Jul 08 20:17:39.686222 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:17:39.686206 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf\": container with ID starting with 88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf not found: ID does not exist" containerID="88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf" Jul 08 20:17:39.686271 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.686231 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf"} err="failed to get container status \"88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf\": rpc error: code = NotFound desc = could not find container \"88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf\": container with ID starting with 88d170d744e0efbc60ec1a41f21643fe8e94db16bf8afa6598215ae020242acf not found: ID does not exist" Jul 08 20:17:39.759000 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.758978 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rnng\" (UniqueName: \"kubernetes.io/projected/d6006db9-095c-4670-b632-58c6f8127e1c-kube-api-access-5rnng\") pod \"d6006db9-095c-4670-b632-58c6f8127e1c\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " Jul 08 20:17:39.759107 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.759006 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6006db9-095c-4670-b632-58c6f8127e1c-maas-api-tls\") pod \"d6006db9-095c-4670-b632-58c6f8127e1c\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " Jul 08 20:17:39.759107 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.759025 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6006db9-095c-4670-b632-58c6f8127e1c-openshift-service-ca\") pod \"d6006db9-095c-4670-b632-58c6f8127e1c\" (UID: \"d6006db9-095c-4670-b632-58c6f8127e1c\") " Jul 08 20:17:39.759448 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.759423 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6006db9-095c-4670-b632-58c6f8127e1c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6006db9-095c-4670-b632-58c6f8127e1c" (UID: "d6006db9-095c-4670-b632-58c6f8127e1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:17:39.760971 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.760950 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6006db9-095c-4670-b632-58c6f8127e1c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6006db9-095c-4670-b632-58c6f8127e1c" (UID: "d6006db9-095c-4670-b632-58c6f8127e1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:17:39.760971 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.760961 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6006db9-095c-4670-b632-58c6f8127e1c-kube-api-access-5rnng" (OuterVolumeSpecName: "kube-api-access-5rnng") pod "d6006db9-095c-4670-b632-58c6f8127e1c" (UID: "d6006db9-095c-4670-b632-58c6f8127e1c"). InnerVolumeSpecName "kube-api-access-5rnng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:17:39.859758 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.859736 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rnng\" (UniqueName: \"kubernetes.io/projected/d6006db9-095c-4670-b632-58c6f8127e1c-kube-api-access-5rnng\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:39.859758 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.859757 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6006db9-095c-4670-b632-58c6f8127e1c-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:39.859886 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.859767 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6006db9-095c-4670-b632-58c6f8127e1c-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:39.999233 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:39.999204 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj"] Jul 08 20:17:40.004115 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:40.004093 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-db100e-5cc96fc864-s7rwj"] Jul 08 20:17:41.673767 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:41.673732 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6006db9-095c-4670-b632-58c6f8127e1c" path="/var/lib/kubelet/pods/d6006db9-095c-4670-b632-58c6f8127e1c/volumes" Jul 08 20:17:42.990686 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:42.990649 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:17:52.595885 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.595820 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" podUID="f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" containerName="main" containerID="cri-o://929dbfda79a8b82f8457b76721a5874854faa340e17294b749291afc013b57b8" gracePeriod=45 Jul 08 20:17:52.743574 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.743543 2572 generic.go:356] "Generic (PLEG): container finished" podID="f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" containerID="929dbfda79a8b82f8457b76721a5874854faa340e17294b749291afc013b57b8" exitCode=0 Jul 08 20:17:52.743739 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.743599 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" event={"ID":"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2","Type":"ContainerDied","Data":"929dbfda79a8b82f8457b76721a5874854faa340e17294b749291afc013b57b8"} Jul 08 20:17:52.845005 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.844978 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:17:52.970536 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.970465 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-model-cache\") pod \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " Jul 08 20:17:52.970536 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.970516 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tmp-dir\") pod \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " Jul 08 20:17:52.970536 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.970541 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tls-certs\") pod \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " Jul 08 20:17:52.970735 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.970600 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-dshm\") pod \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " Jul 08 20:17:52.970735 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.970629 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gflfp\" (UniqueName: \"kubernetes.io/projected/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-kube-api-access-gflfp\") pod \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " Jul 08 20:17:52.970735 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.970662 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-home\") pod \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\" (UID: \"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2\") " Jul 08 20:17:52.970735 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.970707 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" (UID: "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:17:52.970923 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.970825 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" (UID: "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:17:52.970980 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.970947 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tmp-dir\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:52.970980 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.970963 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-model-cache\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:52.971071 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.971010 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-home" (OuterVolumeSpecName: "home") pod "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" (UID: "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:17:52.972801 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.972772 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-dshm" (OuterVolumeSpecName: "dshm") pod "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" (UID: "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:17:52.972913 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.972806 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" (UID: "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:17:52.972913 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:52.972846 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-kube-api-access-gflfp" (OuterVolumeSpecName: "kube-api-access-gflfp") pod "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" (UID: "f08e93dc-796c-4378-8e8e-b8bb29a7e9c2"). InnerVolumeSpecName "kube-api-access-gflfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:17:53.071312 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.071291 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-dshm\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:53.071312 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.071311 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gflfp\" (UniqueName: \"kubernetes.io/projected/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-kube-api-access-gflfp\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:53.071439 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.071320 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-home\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:53.071439 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.071328 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2-tls-certs\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:53.526964 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.526926 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" podUID="f1a1f911-6ac9-4d6a-97e7-46987634e1fa" containerName="main" containerID="cri-o://c7f37f878907384fdf4b5645b24dc7e871317eb51bda2a79ac29d98562c4963b" gracePeriod=45 Jul 08 20:17:53.751138 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.751112 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" Jul 08 20:17:53.751527 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.751119 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l" event={"ID":"f08e93dc-796c-4378-8e8e-b8bb29a7e9c2","Type":"ContainerDied","Data":"bc1a704b4783978a783070b3de337df6a80c5d98c3887752fc5d0ae7060b9680"} Jul 08 20:17:53.751527 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.751232 2572 scope.go:117] "RemoveContainer" containerID="929dbfda79a8b82f8457b76721a5874854faa340e17294b749291afc013b57b8" Jul 08 20:17:53.753014 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.752992 2572 generic.go:356] "Generic (PLEG): container finished" podID="f1a1f911-6ac9-4d6a-97e7-46987634e1fa" containerID="c7f37f878907384fdf4b5645b24dc7e871317eb51bda2a79ac29d98562c4963b" exitCode=0 Jul 08 20:17:53.753125 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.753069 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" event={"ID":"f1a1f911-6ac9-4d6a-97e7-46987634e1fa","Type":"ContainerDied","Data":"c7f37f878907384fdf4b5645b24dc7e871317eb51bda2a79ac29d98562c4963b"} Jul 08 20:17:53.769076 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.769054 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l"] Jul 08 20:17:53.777675 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.777620 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-db100e/test-model-db100e-kserve-69bff44f85-46x4l"] Jul 08 20:17:53.782496 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.782480 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:53.878182 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878138 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-dshm\") pod \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " Jul 08 20:17:53.878303 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878228 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-home\") pod \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " Jul 08 20:17:53.878303 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878256 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-model-cache\") pod \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " Jul 08 20:17:53.878303 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878297 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tls-certs\") pod \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " Jul 08 20:17:53.878461 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878352 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4svr5\" (UniqueName: \"kubernetes.io/projected/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-kube-api-access-4svr5\") pod \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " Jul 08 20:17:53.878461 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878389 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tmp-dir\") pod \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\" (UID: \"f1a1f911-6ac9-4d6a-97e7-46987634e1fa\") " Jul 08 20:17:53.878551 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878494 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-home" (OuterVolumeSpecName: "home") pod "f1a1f911-6ac9-4d6a-97e7-46987634e1fa" (UID: "f1a1f911-6ac9-4d6a-97e7-46987634e1fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:17:53.878551 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878510 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1a1f911-6ac9-4d6a-97e7-46987634e1fa" (UID: "f1a1f911-6ac9-4d6a-97e7-46987634e1fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:17:53.878707 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878682 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1a1f911-6ac9-4d6a-97e7-46987634e1fa" (UID: "f1a1f911-6ac9-4d6a-97e7-46987634e1fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:17:53.878775 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878698 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-home\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:53.878775 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.878733 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-model-cache\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:53.880325 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.880306 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-dshm" (OuterVolumeSpecName: "dshm") pod "f1a1f911-6ac9-4d6a-97e7-46987634e1fa" (UID: "f1a1f911-6ac9-4d6a-97e7-46987634e1fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:17:53.880406 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.880356 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1a1f911-6ac9-4d6a-97e7-46987634e1fa" (UID: "f1a1f911-6ac9-4d6a-97e7-46987634e1fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:17:53.880469 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.880454 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-kube-api-access-4svr5" (OuterVolumeSpecName: "kube-api-access-4svr5") pod "f1a1f911-6ac9-4d6a-97e7-46987634e1fa" (UID: "f1a1f911-6ac9-4d6a-97e7-46987634e1fa"). InnerVolumeSpecName "kube-api-access-4svr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:17:53.979618 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.979585 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-dshm\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:53.979618 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.979616 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tls-certs\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:53.979618 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.979626 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4svr5\" (UniqueName: \"kubernetes.io/projected/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-kube-api-access-4svr5\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:53.979774 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:53.979635 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1a1f911-6ac9-4d6a-97e7-46987634e1fa-tmp-dir\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:17:54.759630 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:54.759601 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" Jul 08 20:17:54.760091 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:54.759648 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9" event={"ID":"f1a1f911-6ac9-4d6a-97e7-46987634e1fa","Type":"ContainerDied","Data":"bdeeed11460705ce52e001a1123c051146f49c6b45b66361c7c24c14215e11ad"} Jul 08 20:17:54.760091 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:54.759698 2572 scope.go:117] "RemoveContainer" containerID="c7f37f878907384fdf4b5645b24dc7e871317eb51bda2a79ac29d98562c4963b" Jul 08 20:17:54.780842 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:54.780810 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9"] Jul 08 20:17:54.784263 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:54.784242 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4eb9fe/test-model-4eb9fe-kserve-577c6589dc-4znz9"] Jul 08 20:17:55.673755 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:55.673719 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" path="/var/lib/kubelet/pods/f08e93dc-796c-4378-8e8e-b8bb29a7e9c2/volumes" Jul 08 20:17:55.674097 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:17:55.674084 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1a1f911-6ac9-4d6a-97e7-46987634e1fa" path="/var/lib/kubelet/pods/f1a1f911-6ac9-4d6a-97e7-46987634e1fa/volumes" Jul 08 20:18:22.305625 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.305587 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m"] Jul 08 20:18:22.306138 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.305821 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" podUID="1744fb44-03eb-4276-8e68-e3ca279747b6" containerName="maas-api" containerID="cri-o://43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda" gracePeriod=30 Jul 08 20:18:22.553543 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.553505 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:18:22.595767 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.595706 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck82l\" (UniqueName: \"kubernetes.io/projected/1744fb44-03eb-4276-8e68-e3ca279747b6-kube-api-access-ck82l\") pod \"1744fb44-03eb-4276-8e68-e3ca279747b6\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " Jul 08 20:18:22.595767 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.595747 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1744fb44-03eb-4276-8e68-e3ca279747b6-openshift-service-ca\") pod \"1744fb44-03eb-4276-8e68-e3ca279747b6\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " Jul 08 20:18:22.595913 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.595869 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1744fb44-03eb-4276-8e68-e3ca279747b6-maas-api-tls\") pod \"1744fb44-03eb-4276-8e68-e3ca279747b6\" (UID: \"1744fb44-03eb-4276-8e68-e3ca279747b6\") " Jul 08 20:18:22.596106 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.596074 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1744fb44-03eb-4276-8e68-e3ca279747b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1744fb44-03eb-4276-8e68-e3ca279747b6" (UID: "1744fb44-03eb-4276-8e68-e3ca279747b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:18:22.597797 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.597772 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1744fb44-03eb-4276-8e68-e3ca279747b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1744fb44-03eb-4276-8e68-e3ca279747b6" (UID: "1744fb44-03eb-4276-8e68-e3ca279747b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:18:22.597874 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.597816 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1744fb44-03eb-4276-8e68-e3ca279747b6-kube-api-access-ck82l" (OuterVolumeSpecName: "kube-api-access-ck82l") pod "1744fb44-03eb-4276-8e68-e3ca279747b6" (UID: "1744fb44-03eb-4276-8e68-e3ca279747b6"). InnerVolumeSpecName "kube-api-access-ck82l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:18:22.696637 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.696610 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1744fb44-03eb-4276-8e68-e3ca279747b6-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:18:22.696637 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.696632 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck82l\" (UniqueName: \"kubernetes.io/projected/1744fb44-03eb-4276-8e68-e3ca279747b6-kube-api-access-ck82l\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:18:22.696858 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.696647 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1744fb44-03eb-4276-8e68-e3ca279747b6-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:18:22.898634 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.898543 2572 generic.go:356] "Generic (PLEG): container finished" podID="1744fb44-03eb-4276-8e68-e3ca279747b6" containerID="43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda" exitCode=0 Jul 08 20:18:22.898634 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.898594 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" Jul 08 20:18:22.898860 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.898632 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" event={"ID":"1744fb44-03eb-4276-8e68-e3ca279747b6","Type":"ContainerDied","Data":"43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda"} Jul 08 20:18:22.898860 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.898696 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m" event={"ID":"1744fb44-03eb-4276-8e68-e3ca279747b6","Type":"ContainerDied","Data":"b8dea17c9b8dfb61b0d25ccddb9d9ad8e55d918f33b902b23dd2cda9d80ce403"} Jul 08 20:18:22.898860 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.898726 2572 scope.go:117] "RemoveContainer" containerID="43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda" Jul 08 20:18:22.908022 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.907995 2572 scope.go:117] "RemoveContainer" containerID="43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda" Jul 08 20:18:22.908306 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:18:22.908283 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda\": container with ID starting with 43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda not found: ID does not exist" containerID="43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda" Jul 08 20:18:22.908383 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.908317 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda"} err="failed to get container status \"43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda\": rpc error: code = NotFound desc = could not find container \"43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda\": container with ID starting with 43797e292fb54f1abe8911a37e8edc6730bc879b4820e3b9c3e81f5dd7ccedda not found: ID does not exist" Jul 08 20:18:22.919144 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.919116 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m"] Jul 08 20:18:22.922588 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:22.922569 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4eb9fe-7df88f7dd-dcs4m"] Jul 08 20:18:23.673396 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:23.673362 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1744fb44-03eb-4276-8e68-e3ca279747b6" path="/var/lib/kubelet/pods/1744fb44-03eb-4276-8e68-e3ca279747b6/volumes" Jul 08 20:18:24.292074 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:24.292037 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:18:31.539533 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539496 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2"] Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539786 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6006db9-095c-4670-b632-58c6f8127e1c" containerName="maas-api" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539796 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6006db9-095c-4670-b632-58c6f8127e1c" containerName="maas-api" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539806 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1744fb44-03eb-4276-8e68-e3ca279747b6" containerName="maas-api" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539811 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1744fb44-03eb-4276-8e68-e3ca279747b6" containerName="maas-api" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539820 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" containerName="main" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539825 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" containerName="main" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539836 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a1f911-6ac9-4d6a-97e7-46987634e1fa" containerName="main" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539841 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a1f911-6ac9-4d6a-97e7-46987634e1fa" containerName="main" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539894 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08e93dc-796c-4378-8e8e-b8bb29a7e9c2" containerName="main" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539902 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a1f911-6ac9-4d6a-97e7-46987634e1fa" containerName="main" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539908 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6006db9-095c-4670-b632-58c6f8127e1c" containerName="maas-api" Jul 08 20:18:31.540006 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.539915 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1744fb44-03eb-4276-8e68-e3ca279747b6" containerName="maas-api" Jul 08 20:18:31.544375 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.544354 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.546537 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.546476 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-85eb30-serving-cert\"" Jul 08 20:18:31.546659 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.546615 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:18:31.546733 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.546678 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:18:31.547310 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.547291 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:18:31.552087 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.552034 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2"] Jul 08 20:18:31.664119 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.664084 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c269180-21de-4a02-9ca0-db3fd2aacdbe-maas-api-tls\") pod \"maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.664292 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.664127 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c269180-21de-4a02-9ca0-db3fd2aacdbe-openshift-service-ca\") pod \"maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.664292 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.664154 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pmm8\" (UniqueName: \"kubernetes.io/projected/7c269180-21de-4a02-9ca0-db3fd2aacdbe-kube-api-access-7pmm8\") pod \"maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.765319 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.765288 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c269180-21de-4a02-9ca0-db3fd2aacdbe-openshift-service-ca\") pod \"maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.765469 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.765326 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pmm8\" (UniqueName: \"kubernetes.io/projected/7c269180-21de-4a02-9ca0-db3fd2aacdbe-kube-api-access-7pmm8\") pod \"maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.765469 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.765448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c269180-21de-4a02-9ca0-db3fd2aacdbe-maas-api-tls\") pod \"maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.765982 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.765959 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c269180-21de-4a02-9ca0-db3fd2aacdbe-openshift-service-ca\") pod \"maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.767814 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.767797 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c269180-21de-4a02-9ca0-db3fd2aacdbe-maas-api-tls\") pod \"maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.773674 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.773648 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pmm8\" (UniqueName: \"kubernetes.io/projected/7c269180-21de-4a02-9ca0-db3fd2aacdbe-kube-api-access-7pmm8\") pod \"maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.858679 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.858618 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:31.989510 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:31.989488 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2"] Jul 08 20:18:31.994755 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:18:31.994722 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c269180_21de_4a02_9ca0_db3fd2aacdbe.slice/crio-66c552208d238b619a849602d01c1fd851f7ed8f300c6a18cc371daf93150b46 WatchSource:0}: Error finding container 66c552208d238b619a849602d01c1fd851f7ed8f300c6a18cc371daf93150b46: Status 404 returned error can't find the container with id 66c552208d238b619a849602d01c1fd851f7ed8f300c6a18cc371daf93150b46 Jul 08 20:18:32.953088 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:32.953053 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" event={"ID":"7c269180-21de-4a02-9ca0-db3fd2aacdbe","Type":"ContainerStarted","Data":"fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be"} Jul 08 20:18:32.953569 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:32.953108 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" event={"ID":"7c269180-21de-4a02-9ca0-db3fd2aacdbe","Type":"ContainerStarted","Data":"66c552208d238b619a849602d01c1fd851f7ed8f300c6a18cc371daf93150b46"} Jul 08 20:18:32.953569 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:32.953199 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:32.969075 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:32.969029 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" podStartSLOduration=1.512858262 podStartE2EDuration="1.969015319s" totalImagesPullingTime="456.157057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:18:32.968325607 +0000 UTC m=+2184.132876012" watchObservedRunningTime="2026-07-08 20:18:32.969015319 +0000 UTC m=+2184.133565722" Jul 08 20:18:38.962834 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:38.962806 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:39.310395 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.310363 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:18:39.548633 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.548598 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr"] Jul 08 20:18:39.552580 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.552552 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.554894 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.554869 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-27cd6d-serving-cert\"" Jul 08 20:18:39.565210 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.565134 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr"] Jul 08 20:18:39.622335 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.622290 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xnp\" (UniqueName: \"kubernetes.io/projected/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-kube-api-access-98xnp\") pod \"maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.622495 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.622375 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-openshift-service-ca\") pod \"maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.622495 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.622441 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-maas-api-tls\") pod \"maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.723305 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.723266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-openshift-service-ca\") pod \"maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.723479 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.723330 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-maas-api-tls\") pod \"maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.723479 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.723369 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xnp\" (UniqueName: \"kubernetes.io/projected/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-kube-api-access-98xnp\") pod \"maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.723981 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.723961 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-openshift-service-ca\") pod \"maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.725777 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.725757 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-maas-api-tls\") pod \"maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.730857 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.730833 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xnp\" (UniqueName: \"kubernetes.io/projected/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-kube-api-access-98xnp\") pod \"maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.863319 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.863262 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:39.993696 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:39.993674 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr"] Jul 08 20:18:39.995428 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:18:39.995406 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5ccb6f_ca17_42b1_ba65_ba7b5a37fb8d.slice/crio-d4d00c93e0b161a0745369347d5fab4ce67a925a1ffe48163e211b1f06fa6bc3 WatchSource:0}: Error finding container d4d00c93e0b161a0745369347d5fab4ce67a925a1ffe48163e211b1f06fa6bc3: Status 404 returned error can't find the container with id d4d00c93e0b161a0745369347d5fab4ce67a925a1ffe48163e211b1f06fa6bc3 Jul 08 20:18:40.992752 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:40.992670 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" event={"ID":"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d","Type":"ContainerStarted","Data":"affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081"} Jul 08 20:18:40.992752 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:40.992721 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" event={"ID":"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d","Type":"ContainerStarted","Data":"d4d00c93e0b161a0745369347d5fab4ce67a925a1ffe48163e211b1f06fa6bc3"} Jul 08 20:18:40.992921 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:40.992766 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:41.009345 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:41.009299 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" podStartSLOduration=1.473848599 podStartE2EDuration="2.009285034s" totalImagesPullingTime="535.436435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:18:41.00732422 +0000 UTC m=+2192.171874622" watchObservedRunningTime="2026-07-08 20:18:41.009285034 +0000 UTC m=+2192.173835438" Jul 08 20:18:47.003606 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:47.003577 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:52.810031 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:52.809996 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2"] Jul 08 20:18:52.810526 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:52.810301 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" podUID="7c269180-21de-4a02-9ca0-db3fd2aacdbe" containerName="maas-api" containerID="cri-o://fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be" gracePeriod=30 Jul 08 20:18:53.050091 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.050066 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:53.054036 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.054008 2572 generic.go:356] "Generic (PLEG): container finished" podID="7c269180-21de-4a02-9ca0-db3fd2aacdbe" containerID="fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be" exitCode=0 Jul 08 20:18:53.054241 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.054045 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" Jul 08 20:18:53.054241 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.054100 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" event={"ID":"7c269180-21de-4a02-9ca0-db3fd2aacdbe","Type":"ContainerDied","Data":"fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be"} Jul 08 20:18:53.054241 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.054150 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2" event={"ID":"7c269180-21de-4a02-9ca0-db3fd2aacdbe","Type":"ContainerDied","Data":"66c552208d238b619a849602d01c1fd851f7ed8f300c6a18cc371daf93150b46"} Jul 08 20:18:53.054241 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.054181 2572 scope.go:117] "RemoveContainer" containerID="fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be" Jul 08 20:18:53.062434 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.062364 2572 scope.go:117] "RemoveContainer" containerID="fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be" Jul 08 20:18:53.062679 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:18:53.062656 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be\": container with ID starting with fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be not found: ID does not exist" containerID="fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be" Jul 08 20:18:53.062750 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.062687 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be"} err="failed to get container status \"fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be\": rpc error: code = NotFound desc = could not find container \"fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be\": container with ID starting with fe10f41a085e070573963d61ed10340f067f7fa91096a2f25c1f8be64c3297be not found: ID does not exist" Jul 08 20:18:53.132457 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.132429 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c269180-21de-4a02-9ca0-db3fd2aacdbe-openshift-service-ca\") pod \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " Jul 08 20:18:53.132672 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.132481 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c269180-21de-4a02-9ca0-db3fd2aacdbe-maas-api-tls\") pod \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " Jul 08 20:18:53.132672 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.132504 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pmm8\" (UniqueName: \"kubernetes.io/projected/7c269180-21de-4a02-9ca0-db3fd2aacdbe-kube-api-access-7pmm8\") pod \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\" (UID: \"7c269180-21de-4a02-9ca0-db3fd2aacdbe\") " Jul 08 20:18:53.132845 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.132819 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c269180-21de-4a02-9ca0-db3fd2aacdbe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c269180-21de-4a02-9ca0-db3fd2aacdbe" (UID: "7c269180-21de-4a02-9ca0-db3fd2aacdbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:18:53.134597 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.134575 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c269180-21de-4a02-9ca0-db3fd2aacdbe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c269180-21de-4a02-9ca0-db3fd2aacdbe" (UID: "7c269180-21de-4a02-9ca0-db3fd2aacdbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:18:53.134701 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.134649 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c269180-21de-4a02-9ca0-db3fd2aacdbe-kube-api-access-7pmm8" (OuterVolumeSpecName: "kube-api-access-7pmm8") pod "7c269180-21de-4a02-9ca0-db3fd2aacdbe" (UID: "7c269180-21de-4a02-9ca0-db3fd2aacdbe"). InnerVolumeSpecName "kube-api-access-7pmm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:18:53.233632 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.233604 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c269180-21de-4a02-9ca0-db3fd2aacdbe-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:18:53.233632 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.233629 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c269180-21de-4a02-9ca0-db3fd2aacdbe-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:18:53.233632 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.233639 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pmm8\" (UniqueName: \"kubernetes.io/projected/7c269180-21de-4a02-9ca0-db3fd2aacdbe-kube-api-access-7pmm8\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:18:53.375211 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.375181 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2"] Jul 08 20:18:53.377074 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.377055 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-85eb30-fbb6dd75d-kfht2"] Jul 08 20:18:53.674430 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:53.674355 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c269180-21de-4a02-9ca0-db3fd2aacdbe" path="/var/lib/kubelet/pods/7c269180-21de-4a02-9ca0-db3fd2aacdbe/volumes" Jul 08 20:18:57.087096 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:57.087056 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:18:59.336446 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.336411 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr"] Jul 08 20:18:59.336900 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.336669 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" podUID="5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d" containerName="maas-api" containerID="cri-o://affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081" gracePeriod=30 Jul 08 20:18:59.577461 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.577437 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:18:59.687142 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.687062 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-maas-api-tls\") pod \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " Jul 08 20:18:59.687142 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.687121 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98xnp\" (UniqueName: \"kubernetes.io/projected/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-kube-api-access-98xnp\") pod \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " Jul 08 20:18:59.687338 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.687144 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-openshift-service-ca\") pod \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\" (UID: \"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d\") " Jul 08 20:18:59.687559 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.687537 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d" (UID: "5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:18:59.689200 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.689134 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d" (UID: "5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:18:59.689200 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.689185 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-kube-api-access-98xnp" (OuterVolumeSpecName: "kube-api-access-98xnp") pod "5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d" (UID: "5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d"). InnerVolumeSpecName "kube-api-access-98xnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:18:59.788266 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.788233 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98xnp\" (UniqueName: \"kubernetes.io/projected/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-kube-api-access-98xnp\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:18:59.788266 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.788259 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:18:59.788266 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:18:59.788270 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:19:00.090938 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:00.090898 2572 generic.go:356] "Generic (PLEG): container finished" podID="5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d" containerID="affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081" exitCode=0 Jul 08 20:19:00.091103 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:00.090939 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" event={"ID":"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d","Type":"ContainerDied","Data":"affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081"} Jul 08 20:19:00.091103 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:00.090944 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" Jul 08 20:19:00.091103 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:00.090992 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr" event={"ID":"5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d","Type":"ContainerDied","Data":"d4d00c93e0b161a0745369347d5fab4ce67a925a1ffe48163e211b1f06fa6bc3"} Jul 08 20:19:00.091103 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:00.091021 2572 scope.go:117] "RemoveContainer" containerID="affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081" Jul 08 20:19:00.100965 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:00.100948 2572 scope.go:117] "RemoveContainer" containerID="affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081" Jul 08 20:19:00.101204 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:19:00.101186 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081\": container with ID starting with affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081 not found: ID does not exist" containerID="affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081" Jul 08 20:19:00.101256 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:00.101210 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081"} err="failed to get container status \"affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081\": rpc error: code = NotFound desc = could not find container \"affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081\": container with ID starting with affbc093c927c8344c69479611e41c5ac17807e03367f8e928e54095828c3081 not found: ID does not exist" Jul 08 20:19:00.112720 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:00.112698 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr"] Jul 08 20:19:00.121125 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:00.121101 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27cd6d-5f55fd8788-9xzfr"] Jul 08 20:19:01.673462 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:01.673427 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d" path="/var/lib/kubelet/pods/5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d/volumes" Jul 08 20:19:09.888425 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.888391 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc"] Jul 08 20:19:09.888915 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.888878 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d" containerName="maas-api" Jul 08 20:19:09.888915 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.888916 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d" containerName="maas-api" Jul 08 20:19:09.889108 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.888932 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c269180-21de-4a02-9ca0-db3fd2aacdbe" containerName="maas-api" Jul 08 20:19:09.889108 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.888940 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c269180-21de-4a02-9ca0-db3fd2aacdbe" containerName="maas-api" Jul 08 20:19:09.889108 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.889030 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5ccb6f-ca17-42b1-ba65-ba7b5a37fb8d" containerName="maas-api" Jul 08 20:19:09.889108 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.889041 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c269180-21de-4a02-9ca0-db3fd2aacdbe" containerName="maas-api" Jul 08 20:19:09.893522 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.893503 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:09.895684 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.895662 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-9caeb3-serving-cert\"" Jul 08 20:19:09.895790 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.895716 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:19:09.895790 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.895758 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:19:09.896360 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.896341 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sb6sh\"" Jul 08 20:19:09.902004 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:09.901696 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc"] Jul 08 20:19:10.070959 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.070927 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:10.071184 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.070964 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:10.071184 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.071115 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgf9n\" (UniqueName: \"kubernetes.io/projected/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-kube-api-access-mgf9n\") pod \"maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:10.172263 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.172232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgf9n\" (UniqueName: \"kubernetes.io/projected/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-kube-api-access-mgf9n\") pod \"maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:10.172418 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.172279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:10.172418 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.172307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:10.172907 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.172879 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:10.174740 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.174721 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-maas-api-tls\") pod \"maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:10.180380 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.180349 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgf9n\" (UniqueName: \"kubernetes.io/projected/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-kube-api-access-mgf9n\") pod \"maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:10.206219 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.206190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:10.335552 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.335525 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc"] Jul 08 20:19:10.337262 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:19:10.337231 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b54f731_2d2d_4559_bf92_5f2fcd87e8fa.slice/crio-ae37ccb06514be14e776c1a037451a58976352282677e0f0f8a41cac9e8509ea WatchSource:0}: Error finding container ae37ccb06514be14e776c1a037451a58976352282677e0f0f8a41cac9e8509ea: Status 404 returned error can't find the container with id ae37ccb06514be14e776c1a037451a58976352282677e0f0f8a41cac9e8509ea Jul 08 20:19:10.698179 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:10.698092 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:19:11.148065 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:11.148030 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" event={"ID":"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa","Type":"ContainerStarted","Data":"21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab"} Jul 08 20:19:11.148448 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:11.148082 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" event={"ID":"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa","Type":"ContainerStarted","Data":"ae37ccb06514be14e776c1a037451a58976352282677e0f0f8a41cac9e8509ea"} Jul 08 20:19:11.148448 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:11.148110 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:11.163623 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:11.163577 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" podStartSLOduration=1.786280188 podStartE2EDuration="2.16356153s" totalImagesPullingTime="377.281342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:19:11.162120784 +0000 UTC m=+2222.326671260" watchObservedRunningTime="2026-07-08 20:19:11.16356153 +0000 UTC m=+2222.328111964" Jul 08 20:19:17.158220 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.158192 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:19:17.792591 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.792560 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft"] Jul 08 20:19:17.795906 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.795880 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:17.797897 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.797872 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-e06d38-serving-cert\"" Jul 08 20:19:17.805483 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.805463 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft"] Jul 08 20:19:17.829407 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.829381 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khs77\" (UniqueName: \"kubernetes.io/projected/cbc891c2-0336-4713-8128-20a62abe891d-kube-api-access-khs77\") pod \"maas-api-e2e-shared-b-e06d38-66d468587-hrvft\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:17.829531 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.829439 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc891c2-0336-4713-8128-20a62abe891d-maas-api-tls\") pod \"maas-api-e2e-shared-b-e06d38-66d468587-hrvft\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:17.829531 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.829515 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc891c2-0336-4713-8128-20a62abe891d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e06d38-66d468587-hrvft\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:17.929872 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.929848 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc891c2-0336-4713-8128-20a62abe891d-maas-api-tls\") pod \"maas-api-e2e-shared-b-e06d38-66d468587-hrvft\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:17.929978 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.929890 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc891c2-0336-4713-8128-20a62abe891d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e06d38-66d468587-hrvft\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:17.929978 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.929917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khs77\" (UniqueName: \"kubernetes.io/projected/cbc891c2-0336-4713-8128-20a62abe891d-kube-api-access-khs77\") pod \"maas-api-e2e-shared-b-e06d38-66d468587-hrvft\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:17.930093 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:19:17.929997 2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-serving-cert: secret "maas-api-e2e-shared-b-e06d38-serving-cert" not found Jul 08 20:19:17.930093 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:19:17.930064 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc891c2-0336-4713-8128-20a62abe891d-maas-api-tls podName:cbc891c2-0336-4713-8128-20a62abe891d nodeName:}" failed. No retries permitted until 2026-07-08 20:19:18.430044715 +0000 UTC m=+2229.594595099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbc891c2-0336-4713-8128-20a62abe891d-maas-api-tls") pod "maas-api-e2e-shared-b-e06d38-66d468587-hrvft" (UID: "cbc891c2-0336-4713-8128-20a62abe891d") : secret "maas-api-e2e-shared-b-e06d38-serving-cert" not found Jul 08 20:19:17.930549 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.930528 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc891c2-0336-4713-8128-20a62abe891d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e06d38-66d468587-hrvft\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:17.939404 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:17.939384 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khs77\" (UniqueName: \"kubernetes.io/projected/cbc891c2-0336-4713-8128-20a62abe891d-kube-api-access-khs77\") pod \"maas-api-e2e-shared-b-e06d38-66d468587-hrvft\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:18.434566 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:18.434529 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc891c2-0336-4713-8128-20a62abe891d-maas-api-tls\") pod \"maas-api-e2e-shared-b-e06d38-66d468587-hrvft\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:18.436831 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:18.436812 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc891c2-0336-4713-8128-20a62abe891d-maas-api-tls\") pod \"maas-api-e2e-shared-b-e06d38-66d468587-hrvft\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:18.710404 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:18.710330 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:18.833500 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:18.833477 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft"] Jul 08 20:19:18.835574 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:19:18.835544 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc891c2_0336_4713_8128_20a62abe891d.slice/crio-9b8d69ddf2c3ad05d85b5953dbbfed3291b9bd45ccd34a99eed0763341b5a870 WatchSource:0}: Error finding container 9b8d69ddf2c3ad05d85b5953dbbfed3291b9bd45ccd34a99eed0763341b5a870: Status 404 returned error can't find the container with id 9b8d69ddf2c3ad05d85b5953dbbfed3291b9bd45ccd34a99eed0763341b5a870 Jul 08 20:19:19.187532 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:19.187502 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" event={"ID":"cbc891c2-0336-4713-8128-20a62abe891d","Type":"ContainerStarted","Data":"9b8d69ddf2c3ad05d85b5953dbbfed3291b9bd45ccd34a99eed0763341b5a870"} Jul 08 20:19:20.192757 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:20.192711 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" event={"ID":"cbc891c2-0336-4713-8128-20a62abe891d","Type":"ContainerStarted","Data":"872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301"} Jul 08 20:19:20.193171 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:20.192813 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:19:20.209128 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:20.209082 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" podStartSLOduration=2.799457908 podStartE2EDuration="3.20906898s" totalImagesPullingTime="409.611072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:19:20.207138304 +0000 UTC m=+2231.371688706" watchObservedRunningTime="2026-07-08 20:19:20.20906898 +0000 UTC m=+2231.373619381" Jul 08 20:19:22.289671 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:22.289637 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:19:26.203027 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:19:26.203000 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:22:09.680493 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:09.680465 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:22:09.688740 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:09.688717 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:22:27.717189 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:27.717139 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc"] Jul 08 20:22:27.717746 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:27.717374 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" podUID="9b54f731-2d2d-4559-bf92-5f2fcd87e8fa" containerName="maas-api" containerID="cri-o://21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab" gracePeriod=30 Jul 08 20:22:27.969729 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:27.969671 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:22:27.989356 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:27.989325 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-maas-api-tls\") pod \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " Jul 08 20:22:27.989503 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:27.989393 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-openshift-service-ca\") pod \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " Jul 08 20:22:27.989503 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:27.989465 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgf9n\" (UniqueName: \"kubernetes.io/projected/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-kube-api-access-mgf9n\") pod \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\" (UID: \"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa\") " Jul 08 20:22:27.989786 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:27.989756 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b54f731-2d2d-4559-bf92-5f2fcd87e8fa" (UID: "9b54f731-2d2d-4559-bf92-5f2fcd87e8fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:22:27.991544 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:27.991513 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b54f731-2d2d-4559-bf92-5f2fcd87e8fa" (UID: "9b54f731-2d2d-4559-bf92-5f2fcd87e8fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:22:27.991544 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:27.991529 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-kube-api-access-mgf9n" (OuterVolumeSpecName: "kube-api-access-mgf9n") pod "9b54f731-2d2d-4559-bf92-5f2fcd87e8fa" (UID: "9b54f731-2d2d-4559-bf92-5f2fcd87e8fa"). InnerVolumeSpecName "kube-api-access-mgf9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:22:28.091082 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.091039 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:22:28.091082 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.091073 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgf9n\" (UniqueName: \"kubernetes.io/projected/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-kube-api-access-mgf9n\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:22:28.091082 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.091083 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:22:28.103007 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.102977 2572 generic.go:356] "Generic (PLEG): container finished" podID="9b54f731-2d2d-4559-bf92-5f2fcd87e8fa" containerID="21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab" exitCode=0 Jul 08 20:22:28.103146 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.103024 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" Jul 08 20:22:28.103146 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.103040 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" event={"ID":"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa","Type":"ContainerDied","Data":"21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab"} Jul 08 20:22:28.103146 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.103081 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc" event={"ID":"9b54f731-2d2d-4559-bf92-5f2fcd87e8fa","Type":"ContainerDied","Data":"ae37ccb06514be14e776c1a037451a58976352282677e0f0f8a41cac9e8509ea"} Jul 08 20:22:28.103146 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.103101 2572 scope.go:117] "RemoveContainer" containerID="21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab" Jul 08 20:22:28.112805 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.112780 2572 scope.go:117] "RemoveContainer" containerID="21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab" Jul 08 20:22:28.113057 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:22:28.113040 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab\": container with ID starting with 21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab not found: ID does not exist" containerID="21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab" Jul 08 20:22:28.113110 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.113066 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab"} err="failed to get container status \"21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab\": rpc error: code = NotFound desc = could not find container \"21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab\": container with ID starting with 21c6ece745394688d60f9bac09e2abe57574f4b69231ac5a29014c94f350eaab not found: ID does not exist" Jul 08 20:22:28.123940 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.123914 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc"] Jul 08 20:22:28.127691 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:28.127660 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-9caeb3-5b75689ccc-b6ljc"] Jul 08 20:22:29.672581 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:29.672550 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b54f731-2d2d-4559-bf92-5f2fcd87e8fa" path="/var/lib/kubelet/pods/9b54f731-2d2d-4559-bf92-5f2fcd87e8fa/volumes" Jul 08 20:22:29.790507 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:29.790471 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:22:34.818610 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:34.813723 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft"] Jul 08 20:22:34.818610 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:34.814054 2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" podUID="cbc891c2-0336-4713-8128-20a62abe891d" containerName="maas-api" containerID="cri-o://872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301" gracePeriod=30 Jul 08 20:22:35.070375 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.070311 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:22:35.139436 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.139405 2572 generic.go:356] "Generic (PLEG): container finished" podID="cbc891c2-0336-4713-8128-20a62abe891d" containerID="872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301" exitCode=0 Jul 08 20:22:35.139620 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.139450 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" Jul 08 20:22:35.139620 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.139486 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" event={"ID":"cbc891c2-0336-4713-8128-20a62abe891d","Type":"ContainerDied","Data":"872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301"} Jul 08 20:22:35.139620 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.139539 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft" event={"ID":"cbc891c2-0336-4713-8128-20a62abe891d","Type":"ContainerDied","Data":"9b8d69ddf2c3ad05d85b5953dbbfed3291b9bd45ccd34a99eed0763341b5a870"} Jul 08 20:22:35.139620 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.139561 2572 scope.go:117] "RemoveContainer" containerID="872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301" Jul 08 20:22:35.150226 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.150202 2572 scope.go:117] "RemoveContainer" containerID="872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301" Jul 08 20:22:35.150520 ip-10-0-131-22 kubenswrapper[2572]: E0708 20:22:35.150500 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301\": container with ID starting with 872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301 not found: ID does not exist" containerID="872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301" Jul 08 20:22:35.150576 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.150531 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301"} err="failed to get container status \"872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301\": rpc error: code = NotFound desc = could not find container \"872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301\": container with ID starting with 872a94cbf0bce5586810202b458fdd02627823ab7398aff53f650857b6c0a301 not found: ID does not exist" Jul 08 20:22:35.152800 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.152785 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khs77\" (UniqueName: \"kubernetes.io/projected/cbc891c2-0336-4713-8128-20a62abe891d-kube-api-access-khs77\") pod \"cbc891c2-0336-4713-8128-20a62abe891d\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " Jul 08 20:22:35.152862 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.152851 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc891c2-0336-4713-8128-20a62abe891d-maas-api-tls\") pod \"cbc891c2-0336-4713-8128-20a62abe891d\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " Jul 08 20:22:35.152898 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.152889 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc891c2-0336-4713-8128-20a62abe891d-openshift-service-ca\") pod \"cbc891c2-0336-4713-8128-20a62abe891d\" (UID: \"cbc891c2-0336-4713-8128-20a62abe891d\") " Jul 08 20:22:35.153286 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.153267 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc891c2-0336-4713-8128-20a62abe891d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbc891c2-0336-4713-8128-20a62abe891d" (UID: "cbc891c2-0336-4713-8128-20a62abe891d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:22:35.154904 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.154877 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc891c2-0336-4713-8128-20a62abe891d-kube-api-access-khs77" (OuterVolumeSpecName: "kube-api-access-khs77") pod "cbc891c2-0336-4713-8128-20a62abe891d" (UID: "cbc891c2-0336-4713-8128-20a62abe891d"). InnerVolumeSpecName "kube-api-access-khs77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:22:35.154904 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.154901 2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc891c2-0336-4713-8128-20a62abe891d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbc891c2-0336-4713-8128-20a62abe891d" (UID: "cbc891c2-0336-4713-8128-20a62abe891d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:22:35.254218 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.254153 2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc891c2-0336-4713-8128-20a62abe891d-openshift-service-ca\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:22:35.254218 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.254212 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khs77\" (UniqueName: \"kubernetes.io/projected/cbc891c2-0336-4713-8128-20a62abe891d-kube-api-access-khs77\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:22:35.254218 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.254225 2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbc891c2-0336-4713-8128-20a62abe891d-maas-api-tls\") on node \"ip-10-0-131-22.ec2.internal\" DevicePath \"\"" Jul 08 20:22:35.460785 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.460747 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft"] Jul 08 20:22:35.464115 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.464085 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e06d38-66d468587-hrvft"] Jul 08 20:22:35.673868 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:35.673829 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc891c2-0336-4713-8128-20a62abe891d" path="/var/lib/kubelet/pods/cbc891c2-0336-4713-8128-20a62abe891d/volumes" Jul 08 20:22:38.390174 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:38.390121 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nwfgp"] Jul 08 20:22:46.023807 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:46.023779 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-c9zjq_85de8a0b-8359-42c6-aa06-4695f0bf4733/server/0.log" Jul 08 20:22:46.360908 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:46.360830 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795d769c5f-8fssj_9a1f5784-c030-4bb8-85ce-eb8eaa157791/copy-manifests/0.log" Jul 08 20:22:46.373288 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:46.373255 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795d769c5f-8fssj_9a1f5784-c030-4bb8-85ce-eb8eaa157791/manager/0.log" Jul 08 20:22:47.274737 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.274699 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk_8a66ea7e-0861-43ba-bdb6-f5bfd61573cf/util/0.log" Jul 08 20:22:47.280800 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.280770 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk_8a66ea7e-0861-43ba-bdb6-f5bfd61573cf/pull/0.log" Jul 08 20:22:47.286194 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.286173 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk_8a66ea7e-0861-43ba-bdb6-f5bfd61573cf/extract/0.log" Jul 08 20:22:47.417588 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.417559 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94_bc3a6113-f959-4f56-949b-8bac13e57cce/util/0.log" Jul 08 20:22:47.423794 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.423776 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94_bc3a6113-f959-4f56-949b-8bac13e57cce/pull/0.log" Jul 08 20:22:47.429630 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.429609 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94_bc3a6113-f959-4f56-949b-8bac13e57cce/extract/0.log" Jul 08 20:22:47.542277 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.542218 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf_bb1bd3fd-1a07-47e2-8bea-847a456bdaf2/extract/0.log" Jul 08 20:22:47.548134 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.548110 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf_bb1bd3fd-1a07-47e2-8bea-847a456bdaf2/util/0.log" Jul 08 20:22:47.553975 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.553957 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf_bb1bd3fd-1a07-47e2-8bea-847a456bdaf2/pull/0.log" Jul 08 20:22:47.683435 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.683411 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z_9a854af9-aeb2-4f8f-a200-edfb4a972e66/extract/0.log" Jul 08 20:22:47.688824 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.688807 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z_9a854af9-aeb2-4f8f-a200-edfb4a972e66/util/0.log" Jul 08 20:22:47.694343 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:47.694326 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z_9a854af9-aeb2-4f8f-a200-edfb4a972e66/pull/0.log" Jul 08 20:22:48.510296 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:48.510256 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nwfgp_3de14616-662d-4c66-8f32-78cf501c7338/limitador/0.log" Jul 08 20:22:49.203764 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:49.203732 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779dfd48cc-b8ktd_96a2e2d6-aca3-42ac-83f5-7978e902a612/kube-auth-proxy/0.log" Jul 08 20:22:49.777687 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:49.777657 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74d89d68cf-sjvht_43c4d634-0090-4a22-acd1-51deeaa3584e/router/0.log" Jul 08 20:22:50.653705 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:50.653673 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-xppxf_bd24a1cc-7f02-4de6-9877-4cd7bde32dda/main/0.log" Jul 08 20:22:54.985515 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.985487 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-np6j7/must-gather-qkwcn"] Jul 08 20:22:54.987656 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.985816 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b54f731-2d2d-4559-bf92-5f2fcd87e8fa" containerName="maas-api" Jul 08 20:22:54.987656 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.985829 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b54f731-2d2d-4559-bf92-5f2fcd87e8fa" containerName="maas-api" Jul 08 20:22:54.987656 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.985842 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc891c2-0336-4713-8128-20a62abe891d" containerName="maas-api" Jul 08 20:22:54.987656 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.985849 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc891c2-0336-4713-8128-20a62abe891d" containerName="maas-api" Jul 08 20:22:54.987656 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.985907 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b54f731-2d2d-4559-bf92-5f2fcd87e8fa" containerName="maas-api" Jul 08 20:22:54.987656 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.985918 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc891c2-0336-4713-8128-20a62abe891d" containerName="maas-api" Jul 08 20:22:54.988841 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.988820 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-np6j7/must-gather-qkwcn" Jul 08 20:22:54.991084 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.991064 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-np6j7\"/\"openshift-service-ca.crt\"" Jul 08 20:22:54.991205 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.991074 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-np6j7\"/\"kube-root-ca.crt\"" Jul 08 20:22:54.991276 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:54.991206 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-np6j7\"/\"default-dockercfg-q8fwp\"" Jul 08 20:22:55.001043 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:55.001025 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-np6j7/must-gather-qkwcn"] Jul 08 20:22:55.123553 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:55.123525 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjhzr\" (UniqueName: \"kubernetes.io/projected/97493de9-bb94-40be-a1dc-91db1fcf59c4-kube-api-access-rjhzr\") pod \"must-gather-qkwcn\" (UID: \"97493de9-bb94-40be-a1dc-91db1fcf59c4\") " pod="openshift-must-gather-np6j7/must-gather-qkwcn" Jul 08 20:22:55.123709 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:55.123582 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/97493de9-bb94-40be-a1dc-91db1fcf59c4-must-gather-output\") pod \"must-gather-qkwcn\" (UID: \"97493de9-bb94-40be-a1dc-91db1fcf59c4\") " pod="openshift-must-gather-np6j7/must-gather-qkwcn" Jul 08 20:22:55.224422 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:55.224396 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/97493de9-bb94-40be-a1dc-91db1fcf59c4-must-gather-output\") pod \"must-gather-qkwcn\" (UID: \"97493de9-bb94-40be-a1dc-91db1fcf59c4\") " pod="openshift-must-gather-np6j7/must-gather-qkwcn" Jul 08 20:22:55.224562 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:55.224452 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjhzr\" (UniqueName: \"kubernetes.io/projected/97493de9-bb94-40be-a1dc-91db1fcf59c4-kube-api-access-rjhzr\") pod \"must-gather-qkwcn\" (UID: \"97493de9-bb94-40be-a1dc-91db1fcf59c4\") " pod="openshift-must-gather-np6j7/must-gather-qkwcn" Jul 08 20:22:55.224703 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:55.224686 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/97493de9-bb94-40be-a1dc-91db1fcf59c4-must-gather-output\") pod \"must-gather-qkwcn\" (UID: \"97493de9-bb94-40be-a1dc-91db1fcf59c4\") " pod="openshift-must-gather-np6j7/must-gather-qkwcn" Jul 08 20:22:55.234641 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:55.234609 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjhzr\" (UniqueName: \"kubernetes.io/projected/97493de9-bb94-40be-a1dc-91db1fcf59c4-kube-api-access-rjhzr\") pod \"must-gather-qkwcn\" (UID: \"97493de9-bb94-40be-a1dc-91db1fcf59c4\") " pod="openshift-must-gather-np6j7/must-gather-qkwcn" Jul 08 20:22:55.298698 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:55.298676 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-np6j7/must-gather-qkwcn" Jul 08 20:22:55.428641 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:55.428617 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-np6j7/must-gather-qkwcn"] Jul 08 20:22:55.430667 ip-10-0-131-22 kubenswrapper[2572]: W0708 20:22:55.430638 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97493de9_bb94_40be_a1dc_91db1fcf59c4.slice/crio-af0f986dd985d827151ab6f131d92188c5fe4e7e419c5044631bcb135f821380 WatchSource:0}: Error finding container af0f986dd985d827151ab6f131d92188c5fe4e7e419c5044631bcb135f821380: Status 404 returned error can't find the container with id af0f986dd985d827151ab6f131d92188c5fe4e7e419c5044631bcb135f821380 Jul 08 20:22:55.432504 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:55.432487 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 20:22:56.242905 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:56.242867 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-np6j7/must-gather-qkwcn" event={"ID":"97493de9-bb94-40be-a1dc-91db1fcf59c4","Type":"ContainerStarted","Data":"af0f986dd985d827151ab6f131d92188c5fe4e7e419c5044631bcb135f821380"} Jul 08 20:22:57.252663 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:57.252624 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-np6j7/must-gather-qkwcn" event={"ID":"97493de9-bb94-40be-a1dc-91db1fcf59c4","Type":"ContainerStarted","Data":"0b491acb234620e2f1e2d49d3fb82fc55d2e2fb29d77b1f216bef93cd485363c"} Jul 08 20:22:57.253137 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:57.252674 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-np6j7/must-gather-qkwcn" event={"ID":"97493de9-bb94-40be-a1dc-91db1fcf59c4","Type":"ContainerStarted","Data":"e8439ffde3781c60a431dff4c624d8626bb75efc3474796b939426265784de29"} Jul 08 20:22:57.267851 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:57.267795 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-np6j7/must-gather-qkwcn" podStartSLOduration=2.230039602 podStartE2EDuration="3.267771669s" totalImagesPullingTime="1.037732067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:22:57.26746371 +0000 UTC m=+2448.432014119" watchObservedRunningTime="2026-07-08 20:22:57.267771669 +0000 UTC m=+2448.432322075" Jul 08 20:22:58.045922 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:58.045887 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q5dlc_d39c7b30-0ef8-46ad-b195-277e67817f64/global-pull-secret-syncer/0.log" Jul 08 20:22:58.192340 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:58.192307 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v4rbh_f8131696-a0e8-4bda-a4c4-6ec7ea877e19/konnectivity-agent/0.log" Jul 08 20:22:58.234741 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:22:58.234712 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-22.ec2.internal_9ee3def2b49a5eb35498d1cdd8d47242/haproxy/0.log" Jul 08 20:23:02.211596 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.211555 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk_8a66ea7e-0861-43ba-bdb6-f5bfd61573cf/extract/0.log" Jul 08 20:23:02.235966 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.235942 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk_8a66ea7e-0861-43ba-bdb6-f5bfd61573cf/util/0.log" Jul 08 20:23:02.276474 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.276444 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vbqtk_8a66ea7e-0861-43ba-bdb6-f5bfd61573cf/pull/0.log" Jul 08 20:23:02.335213 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.335183 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94_bc3a6113-f959-4f56-949b-8bac13e57cce/extract/0.log" Jul 08 20:23:02.375070 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.375037 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94_bc3a6113-f959-4f56-949b-8bac13e57cce/util/0.log" Jul 08 20:23:02.408802 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.408768 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bkn94_bc3a6113-f959-4f56-949b-8bac13e57cce/pull/0.log" Jul 08 20:23:02.464309 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.464223 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf_bb1bd3fd-1a07-47e2-8bea-847a456bdaf2/extract/0.log" Jul 08 20:23:02.487781 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.487758 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf_bb1bd3fd-1a07-47e2-8bea-847a456bdaf2/util/0.log" Jul 08 20:23:02.516574 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.516543 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73gpqdf_bb1bd3fd-1a07-47e2-8bea-847a456bdaf2/pull/0.log" Jul 08 20:23:02.558497 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.558445 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z_9a854af9-aeb2-4f8f-a200-edfb4a972e66/extract/0.log" Jul 08 20:23:02.586519 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.586487 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z_9a854af9-aeb2-4f8f-a200-edfb4a972e66/util/0.log" Jul 08 20:23:02.608654 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:02.608620 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1lxk9z_9a854af9-aeb2-4f8f-a200-edfb4a972e66/pull/0.log" Jul 08 20:23:03.249445 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:03.249293 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nwfgp_3de14616-662d-4c66-8f32-78cf501c7338/limitador/0.log" Jul 08 20:23:05.436773 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:05.436684 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vdkf9_1ca7b9e1-bc8d-4282-979e-99782cfcec6c/node-exporter/0.log" Jul 08 20:23:05.461835 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:05.461801 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vdkf9_1ca7b9e1-bc8d-4282-979e-99782cfcec6c/kube-rbac-proxy/0.log" Jul 08 20:23:05.484444 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:05.484419 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vdkf9_1ca7b9e1-bc8d-4282-979e-99782cfcec6c/init-textfile/0.log" Jul 08 20:23:05.521696 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:05.521664 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kq5rf_f74755ad-e746-424c-94f4-55df332addd2/kube-rbac-proxy-main/0.log" Jul 08 20:23:05.544542 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:05.544514 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kq5rf_f74755ad-e746-424c-94f4-55df332addd2/kube-rbac-proxy-self/0.log" Jul 08 20:23:05.566344 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:05.566317 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kq5rf_f74755ad-e746-424c-94f4-55df332addd2/openshift-state-metrics/0.log" Jul 08 20:23:07.810087 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:07.810048 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/1.log" Jul 08 20:23:07.817440 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:07.817414 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-l7rg9_d2bf9f7a-9465-4c1c-8a14-4a9d39729d03/console-operator/2.log" Jul 08 20:23:08.065308 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.065224 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt"] Jul 08 20:23:08.069348 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.069318 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.076502 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.076470 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt"] Jul 08 20:23:08.165977 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.165927 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-podres\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.166304 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.166261 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-lib-modules\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.166477 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.166459 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-proc\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.166654 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.166626 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-sys\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.166936 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.166907 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmlbp\" (UniqueName: \"kubernetes.io/projected/bd9444b2-40cb-4f5b-ae51-5ea620572b41-kube-api-access-zmlbp\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.268555 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.268501 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-sys\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.268740 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.268633 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmlbp\" (UniqueName: \"kubernetes.io/projected/bd9444b2-40cb-4f5b-ae51-5ea620572b41-kube-api-access-zmlbp\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.268740 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.268647 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-sys\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.268740 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.268693 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-podres\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.268740 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.268717 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-lib-modules\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.268982 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.268756 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-proc\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.268982 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.268875 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-proc\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.269092 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.269031 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-podres\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.269173 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.269121 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd9444b2-40cb-4f5b-ae51-5ea620572b41-lib-modules\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.276333 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.276304 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmlbp\" (UniqueName: \"kubernetes.io/projected/bd9444b2-40cb-4f5b-ae51-5ea620572b41-kube-api-access-zmlbp\") pod \"perf-node-gather-daemonset-gp9kt\" (UID: \"bd9444b2-40cb-4f5b-ae51-5ea620572b41\") " pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.387911 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.387835 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:08.563859 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:08.562974 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt"] Jul 08 20:23:09.363352 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:09.363314 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" event={"ID":"bd9444b2-40cb-4f5b-ae51-5ea620572b41","Type":"ContainerStarted","Data":"1740c8ea285a079f385a94921a68fbbe49f2001ec21d11730ac9a4cbeea4fc41"} Jul 08 20:23:09.363779 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:09.363372 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" event={"ID":"bd9444b2-40cb-4f5b-ae51-5ea620572b41","Type":"ContainerStarted","Data":"cb8fea27c22c7f771c3623ab653eb8a016406defc0ec19dead2fbd79b93d5089"} Jul 08 20:23:09.363779 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:09.363513 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:09.380894 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:09.380832 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" podStartSLOduration=1.380812479 podStartE2EDuration="1.380812479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:23:09.377542546 +0000 UTC m=+2460.542092953" watchObservedRunningTime="2026-07-08 20:23:09.380812479 +0000 UTC m=+2460.545362863" Jul 08 20:23:10.185536 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:10.185470 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gg7bd_6d3f85dc-1bff-4fd1-940e-216b04a813ce/dns/0.log" Jul 08 20:23:10.231590 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:10.231565 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gg7bd_6d3f85dc-1bff-4fd1-940e-216b04a813ce/kube-rbac-proxy/0.log" Jul 08 20:23:10.331624 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:10.331599 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jx9bx_53cf257b-7924-4126-9711-b432a663371a/dns-node-resolver/0.log" Jul 08 20:23:10.897340 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:10.897303 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65785cfc6b-xgln2_9c72d281-8e1b-41fd-886a-14621c07949f/registry/0.log" Jul 08 20:23:10.965904 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:10.965876 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t7s4w_b75c8d82-d083-4cf2-9094-51739c38d0e5/node-ca/0.log" Jul 08 20:23:11.987117 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:11.987082 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779dfd48cc-b8ktd_96a2e2d6-aca3-42ac-83f5-7978e902a612/kube-auth-proxy/0.log" Jul 08 20:23:12.180759 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:12.180725 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74d89d68cf-sjvht_43c4d634-0090-4a22-acd1-51deeaa3584e/router/0.log" Jul 08 20:23:12.861134 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:12.861109 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fnphv_49fc0d84-1956-411d-b080-a249b75458f0/serve-healthcheck-canary/0.log" Jul 08 20:23:13.321455 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:13.321424 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-ll9pj_b68237ef-7e03-4ded-a8e2-3e69518b6736/insights-operator/0.log" Jul 08 20:23:13.321893 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:13.321873 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-ll9pj_b68237ef-7e03-4ded-a8e2-3e69518b6736/insights-operator/1.log" Jul 08 20:23:13.479661 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:13.479630 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srckk_6ea71571-304d-4d37-83fa-50bf418aab60/kube-rbac-proxy/0.log" Jul 08 20:23:13.499955 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:13.499930 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srckk_6ea71571-304d-4d37-83fa-50bf418aab60/exporter/0.log" Jul 08 20:23:13.522549 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:13.522519 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-srckk_6ea71571-304d-4d37-83fa-50bf418aab60/extractor/0.log" Jul 08 20:23:15.379755 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:15.379727 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-np6j7/perf-node-gather-daemonset-gp9kt" Jul 08 20:23:15.778753 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:15.778723 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-c9zjq_85de8a0b-8359-42c6-aa06-4695f0bf4733/server/0.log" Jul 08 20:23:15.979991 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:15.979952 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795d769c5f-8fssj_9a1f5784-c030-4bb8-85ce-eb8eaa157791/manager/0.log" Jul 08 20:23:15.999860 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:15.999828 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-795d769c5f-8fssj_9a1f5784-c030-4bb8-85ce-eb8eaa157791/copy-manifests/0.log" Jul 08 20:23:17.349655 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:17.349543 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cffbd996-hql4p_f23d8219-e7b2-4283-ad02-786e694e1aad/manager/0.log" Jul 08 20:23:17.402977 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:17.402948 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-798qs_4d8f89a4-6bbf-4ef3-a5c7-41f875de9adf/openshift-lws-operator/0.log" Jul 08 20:23:18.454212 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:18.454185 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779dfd48cc-b8ktd_96a2e2d6-aca3-42ac-83f5-7978e902a612/kube-auth-proxy/0.log" Jul 08 20:23:18.596866 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:18.596835 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74d89d68cf-sjvht_43c4d634-0090-4a22-acd1-51deeaa3584e/router/0.log" Jul 08 20:23:19.852456 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:19.852429 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-779dfd48cc-b8ktd_96a2e2d6-aca3-42ac-83f5-7978e902a612/kube-auth-proxy/0.log" Jul 08 20:23:20.008182 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:20.008092 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-74d89d68cf-sjvht_43c4d634-0090-4a22-acd1-51deeaa3584e/router/0.log" Jul 08 20:23:22.847363 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:22.847326 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-x4b9z_2e4617b8-a240-411a-a380-0408e88ed3f1/kube-storage-version-migrator-operator/1.log" Jul 08 20:23:22.848231 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:22.848201 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-x4b9z_2e4617b8-a240-411a-a380-0408e88ed3f1/kube-storage-version-migrator-operator/0.log" Jul 08 20:23:23.776750 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:23.776723 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-852tk_392e7cb3-646e-4452-bf3c-0e44dae9bbff/kube-multus-additional-cni-plugins/0.log" Jul 08 20:23:23.797653 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:23.797624 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-852tk_392e7cb3-646e-4452-bf3c-0e44dae9bbff/egress-router-binary-copy/0.log" Jul 08 20:23:23.817581 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:23.817553 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-852tk_392e7cb3-646e-4452-bf3c-0e44dae9bbff/cni-plugins/0.log" Jul 08 20:23:23.841302 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:23.841276 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-852tk_392e7cb3-646e-4452-bf3c-0e44dae9bbff/bond-cni-plugin/0.log" Jul 08 20:23:23.862902 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:23.862875 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-852tk_392e7cb3-646e-4452-bf3c-0e44dae9bbff/routeoverride-cni/0.log" Jul 08 20:23:23.884927 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:23.884904 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-852tk_392e7cb3-646e-4452-bf3c-0e44dae9bbff/whereabouts-cni-bincopy/0.log" Jul 08 20:23:23.906047 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:23.906017 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-852tk_392e7cb3-646e-4452-bf3c-0e44dae9bbff/whereabouts-cni/0.log" Jul 08 20:23:24.243089 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:24.243012 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cwzwf_83d6a6ae-387e-4c21-94af-9566defef5ca/kube-multus/0.log" Jul 08 20:23:24.366547 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:24.366517 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pr9w2_cc99e320-6751-4034-9446-70884569cdbb/network-metrics-daemon/0.log" Jul 08 20:23:24.384179 ip-10-0-131-22 kubenswrapper[2572]: I0708 20:23:24.384134 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pr9w2_cc99e320-6751-4034-9446-70884569cdbb/kube-rbac-proxy/0.log"