Jul 24 18:43:23.733974 ip-10-0-134-30 systemd[1]: Starting Kubernetes Kubelet... Jul 24 18:43:24.182077 ip-10-0-134-30 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 18:43:24.182077 ip-10-0-134-30 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 18:43:24.182077 ip-10-0-134-30 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 18:43:24.182077 ip-10-0-134-30 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 18:43:24.182077 ip-10-0-134-30 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 18:43:24.183153 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.183038 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 18:43:24.189133 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189117 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189134 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189139 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189144 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189147 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189151 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189154 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189157 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189159 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189162 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189165 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189168 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189170 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189173 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189176 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189178 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189181 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189183 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189186 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189189 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 18:43:24.189210 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189191 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189194 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189196 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189200 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189205 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189208 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189211 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189214 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189216 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189219 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189222 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189225 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189227 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189230 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189232 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189235 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189238 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189240 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189242 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 18:43:24.189685 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189245 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189247 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189250 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189252 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189254 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189257 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189259 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189263 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189265 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189268 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189270 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189272 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189275 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189277 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189281 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189284 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189287 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189290 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189292 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189295 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 18:43:24.190170 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189298 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189300 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189304 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189307 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189310 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189313 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189316 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189319 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189321 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189324 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189326 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189330 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189333 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189336 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189338 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189341 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189344 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189347 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189349 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 18:43:24.190781 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189352 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189354 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189356 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189359 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189362 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189364 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189367 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189369 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189373 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189375 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189378 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189380 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189383 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189385 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189388 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189390 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189393 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189395 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189398 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 18:43:24.191240 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189400 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 18:43:24.191704 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.189403 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 18:43:24.191886 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191873 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 18:43:24.191886 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191885 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191889 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191892 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191894 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191897 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191900 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191902 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191905 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191909 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191912 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191915 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191917 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191920 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191922 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191925 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191928 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191930 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191932 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191935 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191937 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 18:43:24.191940 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191940 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191943 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191946 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191949 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191951 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191954 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191958 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191962 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191964 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191967 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191970 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191974 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191977 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191980 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191982 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191985 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191987 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191990 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191992 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 18:43:24.192437 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191995 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.191998 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192000 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192003 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192006 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192009 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192011 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192014 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192017 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192019 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192021 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192024 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192026 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192029 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192031 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192034 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192037 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192039 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192041 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192044 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 18:43:24.192930 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192046 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192049 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192051 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192054 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192057 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192059 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192062 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192064 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192067 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192069 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192072 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192074 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192077 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192080 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192083 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192087 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192089 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192092 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 18:43:24.193419 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192096 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192099 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192102 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192105 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192108 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192110 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192113 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192115 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192118 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192120 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192123 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192125 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192128 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192130 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192133 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192135 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192138 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192140 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192143 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192145 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 18:43:24.193876 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192148 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192552 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192557 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192560 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192563 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192567 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192569 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192572 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192575 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192578 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192580 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192583 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192586 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192590 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192593 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192596 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192598 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192601 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192603 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192606 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 18:43:24.194374 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192609 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192611 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192614 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192616 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192619 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192621 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192624 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192627 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192629 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192632 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192634 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192637 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192640 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192644 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192647 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192649 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192652 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192654 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192657 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192659 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 18:43:24.194890 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192662 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192664 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192670 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192672 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192675 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192679 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192682 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192684 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192705 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192709 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192713 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192716 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192719 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192722 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192725 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192727 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192730 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192732 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192735 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 18:43:24.195464 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192737 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192740 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192744 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192747 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192750 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192753 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192756 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192760 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192763 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192766 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192768 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192771 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192774 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192783 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192786 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192789 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192798 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192801 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192805 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 18:43:24.195953 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192807 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192811 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192813 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192816 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192818 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192821 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192824 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192826 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192829 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192831 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192833 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192836 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192838 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192841 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192843 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192846 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192848 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192851 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192853 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192856 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 18:43:24.196453 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192858 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.192861 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192929 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192936 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192947 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192952 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192955 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192959 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192972 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192976 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192980 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192984 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192988 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192991 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192994 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.192997 2575 flags.go:64] FLAG: --cgroup-root="" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193000 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193003 2575 flags.go:64] FLAG: --client-ca-file="" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193006 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193009 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193013 2575 flags.go:64] FLAG: --cluster-domain="" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193015 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193018 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193021 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 18:43:24.196942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193024 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193028 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193031 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193034 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193037 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193040 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193043 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193045 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193048 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193051 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193055 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193058 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193061 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193064 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193067 2575 flags.go:64] FLAG: --enable-server="true" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193070 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193073 2575 flags.go:64] FLAG: --event-burst="100" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193076 2575 flags.go:64] FLAG: --event-qps="50" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193079 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193083 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193086 2575 flags.go:64] FLAG: --eviction-hard="" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193089 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193092 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193095 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193098 2575 flags.go:64] FLAG: --eviction-soft="" Jul 24 18:43:24.197541 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193101 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193103 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193106 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193109 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193112 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193114 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193117 2575 flags.go:64] FLAG: --feature-gates="" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193121 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193123 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193126 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193129 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193132 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193134 2575 flags.go:64] FLAG: --help="false" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193137 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193140 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193143 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193146 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193149 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193153 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193156 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193158 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193161 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193164 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 18:43:24.198179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193166 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193169 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193172 2575 flags.go:64] FLAG: --kube-reserved="" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193175 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193178 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193181 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193184 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193186 2575 flags.go:64] FLAG: --lock-file="" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193189 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193192 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193196 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193202 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193204 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193207 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193210 2575 flags.go:64] FLAG: --logging-format="text" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193213 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193216 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193219 2575 flags.go:64] FLAG: --manifest-url="" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193221 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193225 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193228 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193233 2575 flags.go:64] FLAG: --max-pods="110" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193236 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193239 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193242 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 18:43:24.198795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193244 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193247 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193250 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193253 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193259 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193262 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193265 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193268 2575 flags.go:64] FLAG: --pod-cidr="" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193271 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193276 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193278 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193281 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193284 2575 flags.go:64] FLAG: --port="10250" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193287 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193290 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04a2e5a6994a83172" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193293 2575 flags.go:64] FLAG: --qos-reserved="" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193297 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193299 2575 flags.go:64] FLAG: --register-node="true" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193304 2575 flags.go:64] FLAG: --register-with-taints="" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193308 2575 flags.go:64] FLAG: --registry-burst="10" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193311 2575 flags.go:64] FLAG: --registry-qps="5" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193313 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193316 2575 flags.go:64] FLAG: --reserved-memory="" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193320 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193323 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 18:43:24.199405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193326 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193329 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193331 2575 flags.go:64] FLAG: --runonce="false" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193334 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193337 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193340 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193342 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193345 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193349 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193353 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193356 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193358 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193361 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193364 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193367 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193370 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193372 2575 flags.go:64] FLAG: --system-cgroups="" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193375 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193380 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193383 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193386 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193390 2575 flags.go:64] FLAG: --tls-min-version="" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193392 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193395 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193398 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 18:43:24.200067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193402 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193405 2575 flags.go:64] FLAG: --v="2" Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193409 2575 flags.go:64] FLAG: --version="false" Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193412 2575 flags.go:64] FLAG: --vmodule="" Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193416 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.193419 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193532 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193536 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193539 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193541 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193544 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193546 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193549 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193552 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193554 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193559 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193561 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193564 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193567 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193569 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193571 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 18:43:24.200716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193574 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193577 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193579 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193582 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193584 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193586 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193589 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193592 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193594 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193597 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193600 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193604 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193606 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193609 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193612 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193614 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193618 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193622 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193624 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 18:43:24.201269 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193628 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193631 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193634 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193637 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193639 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193642 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193645 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193649 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193652 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193654 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193657 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193659 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193661 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193664 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193667 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193669 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193671 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193674 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193676 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193678 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 24 18:43:24.201744 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193681 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193684 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193701 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193705 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193710 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193714 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193717 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193720 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193723 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193725 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193728 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193731 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193733 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193736 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193739 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193741 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193744 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193747 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193749 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 18:43:24.202245 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193756 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193758 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193762 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193765 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193769 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193771 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193774 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193777 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193779 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193782 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193784 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193787 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193790 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193792 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193795 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193798 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193800 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193804 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193807 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193809 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 18:43:24.202715 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193812 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193815 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193817 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193820 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193823 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.193825 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.194586 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.201959 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.201975 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.201994 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.201999 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202066 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202073 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202077 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 18:43:24.203212 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202080 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202083 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202086 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202090 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202093 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202095 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202098 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202101 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202103 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202106 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202108 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202111 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202114 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202116 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202119 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202121 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202128 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202131 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202134 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 18:43:24.203558 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202136 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202139 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202141 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202144 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202146 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202148 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202151 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202159 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202162 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202164 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202167 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202170 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202172 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202175 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202177 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202181 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202185 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202188 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202190 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 18:43:24.204028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202193 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202196 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202199 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202201 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202204 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202206 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202209 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202211 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202214 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202216 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202219 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202222 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202226 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202229 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202232 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202235 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202237 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202239 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202242 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202244 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 18:43:24.204514 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202247 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202255 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202258 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202260 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202263 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202265 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202268 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202270 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202273 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202275 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202277 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202280 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202282 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202284 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202287 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202289 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202292 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202294 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202296 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202299 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 18:43:24.205007 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202314 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202317 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202320 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202323 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202326 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202328 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202331 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202333 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202336 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202338 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202341 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202343 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202346 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202348 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202356 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202358 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202361 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 24 18:43:24.205493 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202364 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.202368 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202511 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202516 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202519 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202521 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202524 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202527 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202531 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202535 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202538 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202540 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202542 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202545 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 18:43:24.205910 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202547 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202551 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202554 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202557 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202560 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202562 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202565 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202568 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202570 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202573 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202575 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202577 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202580 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202582 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202585 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202587 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202595 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202598 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202601 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202603 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 18:43:24.206246 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202605 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202608 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202610 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202612 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202615 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202617 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202620 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202622 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202624 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202627 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202629 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202632 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202634 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202636 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202639 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202642 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202644 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202646 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202649 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202651 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 18:43:24.206815 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202653 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202656 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202658 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202660 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202663 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202665 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202667 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202670 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202672 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202680 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202683 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202685 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202704 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202708 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202712 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202715 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202718 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202720 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202723 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202725 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 18:43:24.207289 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202728 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202731 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202733 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202735 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202739 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202743 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202746 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202751 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202754 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202757 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202759 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202762 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202764 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202767 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202769 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202772 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202775 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202777 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202780 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 18:43:24.207799 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202782 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202785 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202787 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202796 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202799 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202801 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202804 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:24.202806 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.202811 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.202941 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.205463 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.207146 2575 server.go:1019] "Starting client certificate rotation" Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.207271 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 18:43:24.208258 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.207319 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 18:43:24.210112 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.210096 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 18:43:24.212753 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.212486 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 18:43:24.223417 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.223399 2575 log.go:25] "Validated CRI v1 runtime API" Jul 24 18:43:24.227867 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.227852 2575 log.go:25] "Validated CRI v1 image API" Jul 24 18:43:24.228816 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.228799 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 18:43:24.235187 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.235164 2575 fs.go:135] Filesystem UUIDs: map[3fd922b5-fbe3-4c72-a3f9-b2bf41dfc537:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f157ac99-a56e-454c-b38c-757d0c3dc104:/dev/nvme0n1p4] Jul 24 18:43:24.235291 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.235185 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 18:43:24.240549 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.240434 2575 manager.go:216] Machine: {Timestamp:2026-07-24 18:43:24.239957638 +0000 UTC m=+0.378519799 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094053 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d80b24a1225311aa4e93cb7edfc70 SystemUUID:ec2d80b2-4a12-2531-1aa4-e93cb7edfc70 BootID:613b6dc6-54f1-401b-8cb7-e23ba93bc211 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:98:3b:29:c8:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:98:3b:29:c8:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:0c:bb:a0:42:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 18:43:24.240549 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.240542 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 18:43:24.240668 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.240620 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 18:43:24.240904 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.240886 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 18:43:24.244179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.244150 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 18:43:24.244326 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.244183 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 18:43:24.245077 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.245066 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 18:43:24.245109 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.245080 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 18:43:24.245109 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.245088 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 18:43:24.245970 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.245959 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 18:43:24.247910 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.247900 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 18:43:24.252488 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.252477 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 24 18:43:24.252667 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.252657 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 18:43:24.252716 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.252707 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 24 18:43:24.252750 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.252720 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 18:43:24.252750 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.252732 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 18:43:24.252750 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.252742 2575 kubelet.go:398] "Adding apiserver pod source" Jul 24 18:43:24.252840 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.252750 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 18:43:24.253137 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.253122 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 18:43:24.253197 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.253143 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 18:43:24.255490 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.255475 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 18:43:24.257346 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.257329 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 18:43:24.257421 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.257369 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 18:43:24.258162 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258147 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 18:43:24.258228 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258169 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 18:43:24.258228 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258191 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 18:43:24.258228 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258199 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 18:43:24.258228 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258213 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 18:43:24.258228 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258222 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 18:43:24.258379 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258238 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 18:43:24.258379 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258246 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 18:43:24.258379 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258259 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 18:43:24.258379 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258267 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 18:43:24.258379 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258288 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 18:43:24.258379 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.258295 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 18:43:24.259194 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.259183 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 18:43:24.259223 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.259195 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 18:43:24.262225 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.262200 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 18:43:24.262225 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.262197 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 18:43:24.262395 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.262260 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 18:43:24.262582 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.262571 2575 server.go:1268] "Started kubelet" Jul 24 18:43:24.262730 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.262673 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 18:43:24.262793 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.262655 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 18:43:24.262846 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.262790 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 18:43:24.263496 ip-10-0-134-30 systemd[1]: Started Kubernetes Kubelet. Jul 24 18:43:24.265524 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.265503 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 18:43:24.267158 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.267139 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 24 18:43:24.267587 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.267565 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 18:43:24.276558 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.276538 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 18:43:24.276558 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.276555 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 18:43:24.276771 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.276653 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:24.277271 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.277256 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 18:43:24.277367 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.275547 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-30.ec2.internal.18c54e03800e124d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-30.ec2.internal,UID:ip-10-0-134-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-30.ec2.internal,},FirstTimestamp:2026-07-24 18:43:24.262543949 +0000 UTC m=+0.401106110,LastTimestamp:2026-07-24 18:43:24.262543949 +0000 UTC m=+0.401106110,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-30.ec2.internal,}" Jul 24 18:43:24.277451 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.277309 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 18:43:24.277520 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.277511 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 18:43:24.277639 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.277561 2575 factory.go:55] Registering systemd factory Jul 24 18:43:24.277639 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.277659 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 24 18:43:24.277639 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.277669 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 24 18:43:24.277639 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.277659 2575 factory.go:223] Registration of the systemd container factory successfully Jul 24 18:43:24.277997 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.277977 2575 factory.go:153] Registering CRI-O factory Jul 24 18:43:24.277997 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.277995 2575 factory.go:223] Registration of the crio container factory successfully Jul 24 18:43:24.278167 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.278078 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 18:43:24.278167 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.278105 2575 factory.go:103] Registering Raw factory Jul 24 18:43:24.278167 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.278119 2575 manager.go:1196] Started watching for new ooms in manager Jul 24 18:43:24.278846 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.278832 2575 manager.go:349] Starting recovery of all containers Jul 24 18:43:24.279045 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.279019 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 18:43:24.279248 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.279225 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 18:43:24.288138 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.288118 2575 manager.go:354] Recovery completed Jul 24 18:43:24.292048 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.292035 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 18:43:24.294028 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.293998 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientMemory" Jul 24 18:43:24.294082 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.294042 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 18:43:24.294082 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.294053 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientPID" Jul 24 18:43:24.294554 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.294538 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 18:43:24.294554 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.294552 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 18:43:24.294655 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.294556 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 18:43:24.294655 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.294566 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 24 18:43:24.295951 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.295920 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientMemory" Jul 24 18:43:24.295951 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.295944 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-djtw8" Jul 24 18:43:24.295951 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.295952 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 18:43:24.296113 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.295977 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientPID" Jul 24 18:43:24.296619 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.296033 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-30.ec2.internal.18c54e0381ee7d13 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-30.ec2.internal,UID:ip-10-0-134-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-30.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-30.ec2.internal,},FirstTimestamp:2026-07-24 18:43:24.294028563 +0000 UTC m=+0.432590723,LastTimestamp:2026-07-24 18:43:24.294028563 +0000 UTC m=+0.432590723,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-30.ec2.internal,}" Jul 24 18:43:24.296804 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.296790 2575 policy_none.go:49] "None policy: Start" Jul 24 18:43:24.296860 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.296809 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 18:43:24.296860 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.296821 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 18:43:24.298850 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.298839 2575 policy_none.go:47] "Start" Jul 24 18:43:24.305206 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.305082 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-djtw8" Jul 24 18:43:24.305957 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.305895 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-30.ec2.internal.18c54e0381eec7e0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-30.ec2.internal,UID:ip-10-0-134-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-30.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-30.ec2.internal,},FirstTimestamp:2026-07-24 18:43:24.294047712 +0000 UTC m=+0.432609873,LastTimestamp:2026-07-24 18:43:24.294047712 +0000 UTC m=+0.432609873,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-30.ec2.internal,}" Jul 24 18:43:24.332124 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.332106 2575 manager.go:337] "Starting Device Plugin manager" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.332163 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.332176 2575 server.go:87] "Starting device plugin registration server" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.332432 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.332447 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.332554 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.332829 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.332839 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.333269 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.333326 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.346671 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.347979 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.348013 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.348033 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.348066 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 18:43:24.358233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.349927 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 18:43:24.433564 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.433465 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 18:43:24.434411 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.434390 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientMemory" Jul 24 18:43:24.434527 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.434430 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 18:43:24.434527 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.434443 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientPID" Jul 24 18:43:24.434527 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.434465 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.443025 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.443008 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.443088 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.443028 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-30.ec2.internal\": node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:24.448273 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.448254 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal"] Jul 24 18:43:24.448355 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.448312 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 18:43:24.449126 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.449105 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientMemory" Jul 24 18:43:24.449202 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.449140 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 18:43:24.449202 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.449155 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientPID" Jul 24 18:43:24.451470 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.451456 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 18:43:24.451611 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.451599 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.451647 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.451625 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 18:43:24.452126 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.452099 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientMemory" Jul 24 18:43:24.452195 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.452133 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 18:43:24.452195 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.452145 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientPID" Jul 24 18:43:24.452195 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.452110 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientMemory" Jul 24 18:43:24.452287 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.452206 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 18:43:24.452287 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.452216 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientPID" Jul 24 18:43:24.454280 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.454266 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.454327 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.454291 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 18:43:24.454898 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.454883 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientMemory" Jul 24 18:43:24.454981 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.454913 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 18:43:24.454981 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.454926 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientPID" Jul 24 18:43:24.471968 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.471946 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:24.476261 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.476245 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-30.ec2.internal\" not found" node="ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.478563 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.478546 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e352d90755072d947bfa04f22592e32-config\") pod \"kube-apiserver-proxy-ip-10-0-134-30.ec2.internal\" (UID: \"2e352d90755072d947bfa04f22592e32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.478635 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.478576 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/758f59bb62b76599c47fbe538d30ba69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal\" (UID: \"758f59bb62b76599c47fbe538d30ba69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.478635 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.478606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/758f59bb62b76599c47fbe538d30ba69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal\" (UID: \"758f59bb62b76599c47fbe538d30ba69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.480532 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.480517 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-30.ec2.internal\" not found" node="ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.573019 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.572985 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:24.579350 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.579328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e352d90755072d947bfa04f22592e32-config\") pod \"kube-apiserver-proxy-ip-10-0-134-30.ec2.internal\" (UID: \"2e352d90755072d947bfa04f22592e32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.579402 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.579362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/758f59bb62b76599c47fbe538d30ba69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal\" (UID: \"758f59bb62b76599c47fbe538d30ba69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.579402 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.579382 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/758f59bb62b76599c47fbe538d30ba69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal\" (UID: \"758f59bb62b76599c47fbe538d30ba69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.579488 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.579424 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/758f59bb62b76599c47fbe538d30ba69-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal\" (UID: \"758f59bb62b76599c47fbe538d30ba69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.579488 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.579430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/758f59bb62b76599c47fbe538d30ba69-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal\" (UID: \"758f59bb62b76599c47fbe538d30ba69\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.579488 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.579431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2e352d90755072d947bfa04f22592e32-config\") pod \"kube-apiserver-proxy-ip-10-0-134-30.ec2.internal\" (UID: \"2e352d90755072d947bfa04f22592e32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.673544 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.673505 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:24.774304 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.774240 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:24.779583 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.779557 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.786683 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:24.785454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal" Jul 24 18:43:24.875374 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.875335 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:24.975877 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:24.975845 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:25.076441 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:25.076366 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:25.176844 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:25.176812 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:25.206230 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.206205 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 18:43:25.206740 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.206376 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 18:43:25.277622 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:25.277590 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:25.311019 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.310971 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 18:38:25 +0000 UTC" deadline="2028-03-15 03:28:02.253147112 +0000 UTC" Jul 24 18:43:25.311019 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.311018 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14384h44m36.942134441s" Jul 24 18:43:25.375360 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.375302 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 18:43:25.376176 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.376157 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientMemory" Jul 24 18:43:25.376264 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.376188 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 18:43:25.376264 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.376200 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeHasSufficientPID" Jul 24 18:43:25.378460 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:25.378446 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:25.478678 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:25.478649 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-30.ec2.internal\" not found" Jul 24 18:43:25.519708 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.519667 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 18:43:25.556523 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:25.556492 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e352d90755072d947bfa04f22592e32.slice/crio-4ea9409b6ef41d49411623b08df227eea5627f30fd55f4db9888db66a0fb8453 WatchSource:0}: Error finding container 4ea9409b6ef41d49411623b08df227eea5627f30fd55f4db9888db66a0fb8453: Status 404 returned error can't find the container with id 4ea9409b6ef41d49411623b08df227eea5627f30fd55f4db9888db66a0fb8453 Jul 24 18:43:25.557297 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:25.557269 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758f59bb62b76599c47fbe538d30ba69.slice/crio-2b7a3fcc8e4f269a4e352e665d9ac77bd4b5f4511149a3156686e4c53b961287 WatchSource:0}: Error finding container 2b7a3fcc8e4f269a4e352e665d9ac77bd4b5f4511149a3156686e4c53b961287: Status 404 returned error can't find the container with id 2b7a3fcc8e4f269a4e352e665d9ac77bd4b5f4511149a3156686e4c53b961287 Jul 24 18:43:25.561445 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.561430 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 18:43:25.577791 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.577772 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" Jul 24 18:43:25.593477 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.593457 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 18:43:25.595426 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.595413 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal" Jul 24 18:43:25.604168 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.604155 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 18:43:25.691356 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.691274 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 18:43:25.756812 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:25.756791 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 18:43:27.316218 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.316190 2575 apiserver.go:52] "Watching apiserver" Jul 24 18:43:27.325788 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.325026 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 18:43:27.325788 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.325511 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-dvnrx","kube-system/global-pull-secret-syncer-5q8wz","openshift-cluster-node-tuning-operator/tuned-cb6lh","openshift-dns/node-resolver-pgx49","openshift-image-registry/node-ca-s29nq","openshift-multus/multus-additional-cni-plugins-psph6","openshift-ovn-kubernetes/ovnkube-node-8qzpr","kube-system/konnectivity-agent-rlnzb","kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal","openshift-multus/multus-bfzgn","openshift-multus/network-metrics-daemon-khpvw","openshift-network-diagnostics/network-check-target-x6l9k"] Jul 24 18:43:27.328922 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.328900 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.332780 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.332759 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 18:43:27.332974 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.332958 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 18:43:27.333125 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.333111 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dcjck\"" Jul 24 18:43:27.333258 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.333246 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 18:43:27.333394 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.333381 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 18:43:27.333844 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.333827 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 18:43:27.333924 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.333843 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 18:43:27.334097 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.334082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:27.334198 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.334165 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:27.337948 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.337579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.340128 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.339812 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 18:43:27.340128 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.339914 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 18:43:27.340128 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.339921 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 18:43:27.340128 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.339956 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t7pf4\"" Jul 24 18:43:27.340524 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.340505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.343036 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.342675 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 18:43:27.343036 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.342800 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4w9p8\"" Jul 24 18:43:27.343036 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.342899 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 18:43:27.343036 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.343013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.345143 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.345125 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 18:43:27.345387 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.345369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.345971 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.345677 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 18:43:27.345971 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.345756 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drt4g\"" Jul 24 18:43:27.345971 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.345893 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 18:43:27.348103 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.348086 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 18:43:27.348701 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.348683 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.350945 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351016 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351021 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zf8t7\"" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351094 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351250 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qpz\" (UniqueName: \"kubernetes.io/projected/7d43cff0-e130-45e6-8fbc-e212008b30a4-kube-api-access-s2qpz\") pod \"node-resolver-pgx49\" (UID: \"7d43cff0-e130-45e6-8fbc-e212008b30a4\") " pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351522 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-run-ovn\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-modprobe-d\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-sysctl-d\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-tuned\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-run-netns\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.351773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351775 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-run-systemd\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-var-lib-openvswitch\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-sys\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351860 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-systemd-units\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-cni-bin\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-cni-netd\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrfr\" (UniqueName: \"kubernetes.io/projected/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-kube-api-access-sxrfr\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-var-lib-kubelet\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txhsn\" (UniqueName: \"kubernetes.io/projected/ce6f309d-c9c3-478b-ba2e-d5680d409922-kube-api-access-txhsn\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.351994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d43cff0-e130-45e6-8fbc-e212008b30a4-hosts-file\") pod \"node-resolver-pgx49\" (UID: \"7d43cff0-e130-45e6-8fbc-e212008b30a4\") " pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-ovn-node-metrics-cert\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-ovnkube-script-lib\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-sysconfig\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-systemd\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-lib-modules\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-host\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352129 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-etc-openvswitch\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-log-socket\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352181 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-ovnkube-config\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352201 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-run\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d43cff0-e130-45e6-8fbc-e212008b30a4-tmp-dir\") pod \"node-resolver-pgx49\" (UID: \"7d43cff0-e130-45e6-8fbc-e212008b30a4\") " pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-kubelet\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-run-openvswitch\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352273 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-env-overrides\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce6f309d-c9c3-478b-ba2e-d5680d409922-tmp\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352353 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-slash\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352377 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-kubernetes\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-sysctl-conf\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-node-log\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-run-ovn-kubernetes\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.352823 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.352454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz5ch\" (UniqueName: \"kubernetes.io/projected/3ed67684-a72a-4ec0-9a45-3871b53a3e72-kube-api-access-jz5ch\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:27.353512 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.353182 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:27.353512 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.353325 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:27.353512 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.353402 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 18:43:27.354141 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.353976 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-khfl8\"" Jul 24 18:43:27.354141 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.354060 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 18:43:27.354298 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.353976 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 18:43:27.354690 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.354478 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:27.354690 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.354549 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:27.354690 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.354597 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 18:43:27.365662 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.365466 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.367853 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.367830 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 18:43:27.367935 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.367864 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 18:43:27.367935 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.367899 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 18:43:27.368184 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.368166 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lsnbz\"" Jul 24 18:43:27.369310 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.369287 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:27.370971 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.370950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.371582 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.371562 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-stdbq\"" Jul 24 18:43:27.371844 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.371827 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 18:43:27.372036 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.372017 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 18:43:27.373318 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.373297 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 18:43:27.373414 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.373329 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-77sjv\"" Jul 24 18:43:27.377504 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.377470 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qxdsx" Jul 24 18:43:27.386811 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.386793 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qxdsx" Jul 24 18:43:27.415357 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.415288 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal" event={"ID":"2e352d90755072d947bfa04f22592e32","Type":"ContainerStarted","Data":"4ea9409b6ef41d49411623b08df227eea5627f30fd55f4db9888db66a0fb8453"} Jul 24 18:43:27.417992 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.417963 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" event={"ID":"758f59bb62b76599c47fbe538d30ba69","Type":"ContainerStarted","Data":"2b7a3fcc8e4f269a4e352e665d9ac77bd4b5f4511149a3156686e4c53b961287"} Jul 24 18:43:27.441353 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.441330 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 18:43:27.453112 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453085 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm542\" (UniqueName: \"kubernetes.io/projected/14752100-7a25-4f88-abc5-c6903e11289e-kube-api-access-rm542\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.453239 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-cnibin\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.453239 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453147 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6d3da16-d2a0-4046-b297-759095781b40-multus-daemon-config\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.453239 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-run-ovn\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.453239 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-os-release\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.453396 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453246 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-os-release\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.453396 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-modprobe-d\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.453396 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-tuned\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.453396 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-run-systemd\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.453396 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-var-lib-openvswitch\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.453396 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453349 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14752100-7a25-4f88-abc5-c6903e11289e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.453396 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qwq7\" (UniqueName: \"kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7\") pod \"network-check-target-x6l9k\" (UID: \"641810ec-5d18-41c3-990b-462e721aa175\") " pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:27.453396 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-var-lib-cni-bin\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.453612 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-multus-conf-dir\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.453612 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-sys\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.453612 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-systemd-units\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.453612 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-cni-bin\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.453612 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-cni-netd\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.453612 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:27.453612 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453554 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-etc-kubernetes\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.453612 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6e2711c-af0a-41eb-b4d0-d1d86509fdc1-serviceca\") pod \"node-ca-s29nq\" (UID: \"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1\") " pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xfm\" (UniqueName: \"kubernetes.io/projected/e6e2711c-af0a-41eb-b4d0-d1d86509fdc1-kube-api-access-r5xfm\") pod \"node-ca-s29nq\" (UID: \"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1\") " pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-var-lib-kubelet\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d43cff0-e130-45e6-8fbc-e212008b30a4-hosts-file\") pod \"node-resolver-pgx49\" (UID: \"7d43cff0-e130-45e6-8fbc-e212008b30a4\") " pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-ovnkube-script-lib\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453711 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-multus-cni-dir\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvf4\" (UniqueName: \"kubernetes.io/projected/82e05024-15c2-4996-836d-3bf54836445c-kube-api-access-jqvf4\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-systemd\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-cni-bin\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-cni-netd\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-run-ovn\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-lib-modules\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453906 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-modprobe-d\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-run-systemd\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453951 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-systemd-units\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453970 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7d43cff0-e130-45e6-8fbc-e212008b30a4-hosts-file\") pod \"node-resolver-pgx49\" (UID: \"7d43cff0-e130-45e6-8fbc-e212008b30a4\") " pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453992 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-var-lib-kubelet\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454018 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-sys\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.454054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454018 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-lib-modules\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454035 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-var-lib-openvswitch\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-etc-openvswitch\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-log-socket\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.453946 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-systemd\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.454110 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454135 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-log-socket\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.454180 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs podName:3ed67684-a72a-4ec0-9a45-3871b53a3e72 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:27.954158473 +0000 UTC m=+3.030671027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs") pod "network-metrics-daemon-khpvw" (UID: "3ed67684-a72a-4ec0-9a45-3871b53a3e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-etc-openvswitch\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454257 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-etc-selinux\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454226 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d43cff0-e130-45e6-8fbc-e212008b30a4-tmp-dir\") pod \"node-resolver-pgx49\" (UID: \"7d43cff0-e130-45e6-8fbc-e212008b30a4\") " pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-var-lib-cni-multus\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-slash\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454466 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-slash\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.454926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-cnibin\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6e2711c-af0a-41eb-b4d0-d1d86509fdc1-host\") pod \"node-ca-s29nq\" (UID: \"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1\") " pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-registration-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454582 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-ovnkube-script-lib\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454603 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d43cff0-e130-45e6-8fbc-e212008b30a4-tmp-dir\") pod \"node-resolver-pgx49\" (UID: \"7d43cff0-e130-45e6-8fbc-e212008b30a4\") " pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-kubernetes\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-kubernetes\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz5ch\" (UniqueName: \"kubernetes.io/projected/3ed67684-a72a-4ec0-9a45-3871b53a3e72-kube-api-access-jz5ch\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454708 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-sys-fs\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qpz\" (UniqueName: \"kubernetes.io/projected/7d43cff0-e130-45e6-8fbc-e212008b30a4-kube-api-access-s2qpz\") pod \"node-resolver-pgx49\" (UID: \"7d43cff0-e130-45e6-8fbc-e212008b30a4\") " pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454836 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fee3297a-eb1f-4807-b94b-a8b57a303292-agent-certs\") pod \"konnectivity-agent-rlnzb\" (UID: \"fee3297a-eb1f-4807-b94b-a8b57a303292\") " pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6d3da16-d2a0-4046-b297-759095781b40-cni-binary-copy\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454909 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-run-k8s-cni-cncf-io\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-hostroot\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454960 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-run-multus-certs\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.454989 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-sysctl-d\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.455021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-run-netns\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.455756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.455046 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-run-netns\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.456433 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.455073 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-var-lib-kubelet\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.456433 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.455103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrfr\" (UniqueName: \"kubernetes.io/projected/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-kube-api-access-sxrfr\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.456433 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.455189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14752100-7a25-4f88-abc5-c6903e11289e-cni-binary-copy\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.456433 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.455228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-dbus\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:27.456433 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.455254 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-sysctl-d\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.456433 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.455290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txhsn\" (UniqueName: \"kubernetes.io/projected/ce6f309d-c9c3-478b-ba2e-d5680d409922-kube-api-access-txhsn\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.456433 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.455302 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-run-netns\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.455350 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-ovn-node-metrics-cert\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.457309 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fee3297a-eb1f-4807-b94b-a8b57a303292-konnectivity-ca\") pod \"konnectivity-agent-rlnzb\" (UID: \"fee3297a-eb1f-4807-b94b-a8b57a303292\") " pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.457382 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-system-cni-dir\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.457440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9pn\" (UniqueName: \"kubernetes.io/projected/0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f-kube-api-access-nc9pn\") pod \"iptables-alerter-dvnrx\" (UID: \"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f\") " pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.457655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-sysconfig\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.457869 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-sysconfig\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.457926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-host\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.457987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-ovnkube-config\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-system-cni-dir\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458074 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4662\" (UniqueName: \"kubernetes.io/projected/d6d3da16-d2a0-4046-b297-759095781b40-kube-api-access-q4662\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f-host-slash\") pod \"iptables-alerter-dvnrx\" (UID: \"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f\") " pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458457 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-socket-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-run\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-kubelet\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-run-openvswitch\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458609 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.460894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-env-overrides\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14752100-7a25-4f88-abc5-c6903e11289e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-run\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-kubelet-config\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-tuned\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458812 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-host\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f-iptables-alerter-script\") pod \"iptables-alerter-dvnrx\" (UID: \"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f\") " pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce6f309d-c9c3-478b-ba2e-d5680d409922-tmp\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458929 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-run-openvswitch\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.458958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.459407 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-ovnkube-config\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.459483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-multus-socket-dir-parent\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.459510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-device-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.459542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-sysctl-conf\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.459573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-node-log\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.459605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-run-ovn-kubernetes\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.461700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.460043 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-env-overrides\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.462508 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.460044 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-kubelet\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.462508 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.460161 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-host-run-ovn-kubernetes\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.462508 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.460187 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce6f309d-c9c3-478b-ba2e-d5680d409922-etc-sysctl-conf\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.462508 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.460219 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-node-log\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.463518 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.463492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce6f309d-c9c3-478b-ba2e-d5680d409922-tmp\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.466356 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.466335 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qpz\" (UniqueName: \"kubernetes.io/projected/7d43cff0-e130-45e6-8fbc-e212008b30a4-kube-api-access-s2qpz\") pod \"node-resolver-pgx49\" (UID: \"7d43cff0-e130-45e6-8fbc-e212008b30a4\") " pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.467309 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.467290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz5ch\" (UniqueName: \"kubernetes.io/projected/3ed67684-a72a-4ec0-9a45-3871b53a3e72-kube-api-access-jz5ch\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:27.470974 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.470930 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrfr\" (UniqueName: \"kubernetes.io/projected/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-kube-api-access-sxrfr\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.471253 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.471224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txhsn\" (UniqueName: \"kubernetes.io/projected/ce6f309d-c9c3-478b-ba2e-d5680d409922-kube-api-access-txhsn\") pod \"tuned-cb6lh\" (UID: \"ce6f309d-c9c3-478b-ba2e-d5680d409922\") " pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.476412 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.476351 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/635e0f85-09ee-4cd3-a549-4d5fb3db9b48-ovn-node-metrics-cert\") pod \"ovnkube-node-8qzpr\" (UID: \"635e0f85-09ee-4cd3-a549-4d5fb3db9b48\") " pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.560269 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-run-netns\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.560448 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-var-lib-kubelet\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.560448 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14752100-7a25-4f88-abc5-c6903e11289e-cni-binary-copy\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.560448 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-dbus\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:27.560448 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fee3297a-eb1f-4807-b94b-a8b57a303292-konnectivity-ca\") pod \"konnectivity-agent-rlnzb\" (UID: \"fee3297a-eb1f-4807-b94b-a8b57a303292\") " pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:27.560448 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-system-cni-dir\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.560448 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9pn\" (UniqueName: \"kubernetes.io/projected/0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f-kube-api-access-nc9pn\") pod \"iptables-alerter-dvnrx\" (UID: \"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f\") " pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.560448 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560403 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-run-netns\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-system-cni-dir\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4662\" (UniqueName: \"kubernetes.io/projected/d6d3da16-d2a0-4046-b297-759095781b40-kube-api-access-q4662\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f-host-slash\") pod \"iptables-alerter-dvnrx\" (UID: \"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f\") " pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-socket-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14752100-7a25-4f88-abc5-c6903e11289e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-kubelet-config\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f-iptables-alerter-script\") pod \"iptables-alerter-dvnrx\" (UID: \"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f\") " pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-multus-socket-dir-parent\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560697 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-device-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm542\" (UniqueName: \"kubernetes.io/projected/14752100-7a25-4f88-abc5-c6903e11289e-kube-api-access-rm542\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-cnibin\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.560828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6d3da16-d2a0-4046-b297-759095781b40-multus-daemon-config\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-os-release\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-os-release\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14752100-7a25-4f88-abc5-c6903e11289e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwq7\" (UniqueName: \"kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7\") pod \"network-check-target-x6l9k\" (UID: \"641810ec-5d18-41c3-990b-462e721aa175\") " pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-var-lib-cni-bin\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.560986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-multus-conf-dir\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561029 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-etc-kubernetes\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6e2711c-af0a-41eb-b4d0-d1d86509fdc1-serviceca\") pod \"node-ca-s29nq\" (UID: \"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1\") " pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xfm\" (UniqueName: \"kubernetes.io/projected/e6e2711c-af0a-41eb-b4d0-d1d86509fdc1-kube-api-access-r5xfm\") pod \"node-ca-s29nq\" (UID: \"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1\") " pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561138 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-multus-cni-dir\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvf4\" (UniqueName: \"kubernetes.io/projected/82e05024-15c2-4996-836d-3bf54836445c-kube-api-access-jqvf4\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14752100-7a25-4f88-abc5-c6903e11289e-cni-binary-copy\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561275 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-system-cni-dir\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561336 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.561440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561436 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-dbus\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561520 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f-host-slash\") pod \"iptables-alerter-dvnrx\" (UID: \"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f\") " pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-socket-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fee3297a-eb1f-4807-b94b-a8b57a303292-konnectivity-ca\") pod \"konnectivity-agent-rlnzb\" (UID: \"fee3297a-eb1f-4807-b94b-a8b57a303292\") " pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-system-cni-dir\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561958 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-etc-selinux\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.561980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-var-lib-cni-multus\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-cnibin\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6e2711c-af0a-41eb-b4d0-d1d86509fdc1-host\") pod \"node-ca-s29nq\" (UID: \"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1\") " pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-registration-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-sys-fs\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fee3297a-eb1f-4807-b94b-a8b57a303292-agent-certs\") pod \"konnectivity-agent-rlnzb\" (UID: \"fee3297a-eb1f-4807-b94b-a8b57a303292\") " pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6d3da16-d2a0-4046-b297-759095781b40-cni-binary-copy\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-run-k8s-cni-cncf-io\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-hostroot\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.562166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-run-multus-certs\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.562977 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562216 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/14752100-7a25-4f88-abc5-c6903e11289e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.562977 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562219 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-run-multus-certs\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.562977 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562251 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.562977 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562337 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-var-lib-cni-multus\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.562977 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562312 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-etc-selinux\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.562977 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562365 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-cnibin\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.562977 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562400 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6e2711c-af0a-41eb-b4d0-d1d86509fdc1-host\") pod \"node-ca-s29nq\" (UID: \"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1\") " pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.562977 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562400 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-kubelet-config\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:27.562977 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-registration-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.562977 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.562695 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-run-k8s-cni-cncf-io\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563042 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f-iptables-alerter-script\") pod \"iptables-alerter-dvnrx\" (UID: \"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f\") " pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563109 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-sys-fs\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6d3da16-d2a0-4046-b297-759095781b40-cni-binary-copy\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-var-lib-kubelet\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.563209 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-hostroot\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.563276 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret podName:2a976a8d-b47d-4dbb-a8cd-f9a7170875a8 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:28.063256832 +0000 UTC m=+3.139769382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret") pod "global-pull-secret-syncer-5q8wz" (UID: "2a976a8d-b47d-4dbb-a8cd-f9a7170875a8") : object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563273 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-etc-kubernetes\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563323 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-multus-conf-dir\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-cnibin\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.563438 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-multus-socket-dir-parent\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.563993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563482 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/82e05024-15c2-4996-836d-3bf54836445c-device-dir\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.563993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563543 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/14752100-7a25-4f88-abc5-c6903e11289e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.563993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563729 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-multus-cni-dir\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.563993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563847 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/14752100-7a25-4f88-abc5-c6903e11289e-os-release\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.563993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563860 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e6e2711c-af0a-41eb-b4d0-d1d86509fdc1-serviceca\") pod \"node-ca-s29nq\" (UID: \"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1\") " pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.563993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.563925 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-os-release\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.564229 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.564031 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d6d3da16-d2a0-4046-b297-759095781b40-host-var-lib-cni-bin\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.564332 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.564314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d6d3da16-d2a0-4046-b297-759095781b40-multus-daemon-config\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.564924 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.564903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fee3297a-eb1f-4807-b94b-a8b57a303292-agent-certs\") pod \"konnectivity-agent-rlnzb\" (UID: \"fee3297a-eb1f-4807-b94b-a8b57a303292\") " pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:27.572129 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.572064 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4662\" (UniqueName: \"kubernetes.io/projected/d6d3da16-d2a0-4046-b297-759095781b40-kube-api-access-q4662\") pod \"multus-bfzgn\" (UID: \"d6d3da16-d2a0-4046-b297-759095781b40\") " pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.572315 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.572269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9pn\" (UniqueName: \"kubernetes.io/projected/0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f-kube-api-access-nc9pn\") pod \"iptables-alerter-dvnrx\" (UID: \"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f\") " pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.573341 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.573321 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 18:43:27.573444 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.573347 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 18:43:27.573444 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.573359 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5qwq7 for pod openshift-network-diagnostics/network-check-target-x6l9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:27.573444 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.573401 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7 podName:641810ec-5d18-41c3-990b-462e721aa175 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:28.073390008 +0000 UTC m=+3.149902546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5qwq7" (UniqueName: "kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7") pod "network-check-target-x6l9k" (UID: "641810ec-5d18-41c3-990b-462e721aa175") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:27.577168 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.577103 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvf4\" (UniqueName: \"kubernetes.io/projected/82e05024-15c2-4996-836d-3bf54836445c-kube-api-access-jqvf4\") pod \"aws-ebs-csi-driver-node-zc245\" (UID: \"82e05024-15c2-4996-836d-3bf54836445c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.578232 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.578209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xfm\" (UniqueName: \"kubernetes.io/projected/e6e2711c-af0a-41eb-b4d0-d1d86509fdc1-kube-api-access-r5xfm\") pod \"node-ca-s29nq\" (UID: \"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1\") " pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.580322 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.580292 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm542\" (UniqueName: \"kubernetes.io/projected/14752100-7a25-4f88-abc5-c6903e11289e-kube-api-access-rm542\") pod \"multus-additional-cni-plugins-psph6\" (UID: \"14752100-7a25-4f88-abc5-c6903e11289e\") " pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.646727 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.646693 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:27.665889 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.665858 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" Jul 24 18:43:27.675595 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.675573 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pgx49" Jul 24 18:43:27.681206 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.681181 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s29nq" Jul 24 18:43:27.688848 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.688827 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-psph6" Jul 24 18:43:27.697441 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.697423 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dvnrx" Jul 24 18:43:27.706097 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.706077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" Jul 24 18:43:27.713564 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.713542 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:27.720252 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.720236 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bfzgn" Jul 24 18:43:27.798994 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.798962 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 18:43:27.965100 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:27.965022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:27.965239 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.965174 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:27.965294 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:27.965242 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs podName:3ed67684-a72a-4ec0-9a45-3871b53a3e72 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:28.965223799 +0000 UTC m=+4.041736340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs") pod "network-metrics-daemon-khpvw" (UID: "3ed67684-a72a-4ec0-9a45-3871b53a3e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:28.066068 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:28.066033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:28.066213 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:28.066185 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:28.066284 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:28.066251 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret podName:2a976a8d-b47d-4dbb-a8cd-f9a7170875a8 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:29.06623737 +0000 UTC m=+4.142749906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret") pod "global-pull-secret-syncer-5q8wz" (UID: "2a976a8d-b47d-4dbb-a8cd-f9a7170875a8") : object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:28.166796 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:28.166755 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwq7\" (UniqueName: \"kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7\") pod \"network-check-target-x6l9k\" (UID: \"641810ec-5d18-41c3-990b-462e721aa175\") " pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:28.166963 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:28.166925 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 18:43:28.166963 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:28.166946 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 18:43:28.166963 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:28.166959 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5qwq7 for pod openshift-network-diagnostics/network-check-target-x6l9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:28.167087 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:28.167018 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7 podName:641810ec-5d18-41c3-990b-462e721aa175 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:29.166998206 +0000 UTC m=+4.243510743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qwq7" (UniqueName: "kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7") pod "network-check-target-x6l9k" (UID: "641810ec-5d18-41c3-990b-462e721aa175") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:28.390881 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:28.390793 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 18:38:27 +0000 UTC" deadline="2028-02-02 21:28:35.705058947 +0000 UTC" Jul 24 18:43:28.390881 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:28.390824 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13394h45m7.314239446s" Jul 24 18:43:28.589255 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:28.589227 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee3297a_eb1f_4807_b94b_a8b57a303292.slice/crio-6822a632e0839e77751c9be9f486e06d081493b8918681e85c781ecf2d2e45a4 WatchSource:0}: Error finding container 6822a632e0839e77751c9be9f486e06d081493b8918681e85c781ecf2d2e45a4: Status 404 returned error can't find the container with id 6822a632e0839e77751c9be9f486e06d081493b8918681e85c781ecf2d2e45a4 Jul 24 18:43:28.594185 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:28.594079 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635e0f85_09ee_4cd3_a549_4d5fb3db9b48.slice/crio-2d2a160268f6379d59ffaa90e9789f23c3fa3bb0961d060893e22142f9ff27b3 WatchSource:0}: Error finding container 2d2a160268f6379d59ffaa90e9789f23c3fa3bb0961d060893e22142f9ff27b3: Status 404 returned error can't find the container with id 2d2a160268f6379d59ffaa90e9789f23c3fa3bb0961d060893e22142f9ff27b3 Jul 24 18:43:28.597117 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:28.597090 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e05024_15c2_4996_836d_3bf54836445c.slice/crio-77961d59be7d67e765e9bcb61d5653b869ae161f163301fd0301d1b727097a5a WatchSource:0}: Error finding container 77961d59be7d67e765e9bcb61d5653b869ae161f163301fd0301d1b727097a5a: Status 404 returned error can't find the container with id 77961d59be7d67e765e9bcb61d5653b869ae161f163301fd0301d1b727097a5a Jul 24 18:43:28.600603 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:28.600580 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0484e1e0_0c45_4f9a_a2a8_9882a8dcb14f.slice/crio-263d6f9c78b38ed0974a7e6bf810fe423255a151a14e46023deb2f2db6804ee1 WatchSource:0}: Error finding container 263d6f9c78b38ed0974a7e6bf810fe423255a151a14e46023deb2f2db6804ee1: Status 404 returned error can't find the container with id 263d6f9c78b38ed0974a7e6bf810fe423255a151a14e46023deb2f2db6804ee1 Jul 24 18:43:28.601654 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:28.601563 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d43cff0_e130_45e6_8fbc_e212008b30a4.slice/crio-88b5a3c031f8e700483e3c67cd0cc5b2b34d3953a964a81c8199ec999db8b00f WatchSource:0}: Error finding container 88b5a3c031f8e700483e3c67cd0cc5b2b34d3953a964a81c8199ec999db8b00f: Status 404 returned error can't find the container with id 88b5a3c031f8e700483e3c67cd0cc5b2b34d3953a964a81c8199ec999db8b00f Jul 24 18:43:28.602591 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:28.602365 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14752100_7a25_4f88_abc5_c6903e11289e.slice/crio-2bf67707f9356fe3f05561b0a91516f12ae29a577c610d2dd67f0f303dcc71c9 WatchSource:0}: Error finding container 2bf67707f9356fe3f05561b0a91516f12ae29a577c610d2dd67f0f303dcc71c9: Status 404 returned error can't find the container with id 2bf67707f9356fe3f05561b0a91516f12ae29a577c610d2dd67f0f303dcc71c9 Jul 24 18:43:28.603513 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:43:28.603302 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6f309d_c9c3_478b_ba2e_d5680d409922.slice/crio-1f246cab58e61595f409399d69d17487bd3c5deb47652d7b374b000ed7f87ca4 WatchSource:0}: Error finding container 1f246cab58e61595f409399d69d17487bd3c5deb47652d7b374b000ed7f87ca4: Status 404 returned error can't find the container with id 1f246cab58e61595f409399d69d17487bd3c5deb47652d7b374b000ed7f87ca4 Jul 24 18:43:28.973050 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:28.972864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:28.973189 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:28.973052 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:28.973189 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:28.973134 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs podName:3ed67684-a72a-4ec0-9a45-3871b53a3e72 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:30.973113592 +0000 UTC m=+6.049626143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs") pod "network-metrics-daemon-khpvw" (UID: "3ed67684-a72a-4ec0-9a45-3871b53a3e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:29.073920 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.073877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:29.074314 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:29.074064 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:29.074314 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:29.074140 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret podName:2a976a8d-b47d-4dbb-a8cd-f9a7170875a8 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:31.074108658 +0000 UTC m=+6.150621209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret") pod "global-pull-secret-syncer-5q8wz" (UID: "2a976a8d-b47d-4dbb-a8cd-f9a7170875a8") : object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:29.174804 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.174766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwq7\" (UniqueName: \"kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7\") pod \"network-check-target-x6l9k\" (UID: \"641810ec-5d18-41c3-990b-462e721aa175\") " pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:29.175021 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:29.174978 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 18:43:29.175021 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:29.174999 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 18:43:29.175021 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:29.175012 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5qwq7 for pod openshift-network-diagnostics/network-check-target-x6l9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:29.175213 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:29.175070 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7 podName:641810ec-5d18-41c3-990b-462e721aa175 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:31.175051514 +0000 UTC m=+6.251564065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qwq7" (UniqueName: "kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7") pod "network-check-target-x6l9k" (UID: "641810ec-5d18-41c3-990b-462e721aa175") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:29.391908 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.391777 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 18:38:27 +0000 UTC" deadline="2028-01-10 09:55:33.992067554 +0000 UTC" Jul 24 18:43:29.391908 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.391842 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12831h12m4.600230032s" Jul 24 18:43:29.412574 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.411862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:29.412574 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:29.412003 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:29.412574 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.412438 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:29.412574 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:29.412526 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:29.412917 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.412772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:29.412917 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:29.412867 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:29.449402 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.447814 2575 generic.go:356] "Generic (PLEG): container finished" podID="758f59bb62b76599c47fbe538d30ba69" containerID="4f0c1e658e8631cfb4a8ef9816263318bf60fab9ff7d99536389f711ff468155" exitCode=0 Jul 24 18:43:29.449402 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.448127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" event={"ID":"758f59bb62b76599c47fbe538d30ba69","Type":"ContainerDied","Data":"4f0c1e658e8631cfb4a8ef9816263318bf60fab9ff7d99536389f711ff468155"} Jul 24 18:43:29.473055 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.472987 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal" event={"ID":"2e352d90755072d947bfa04f22592e32","Type":"ContainerStarted","Data":"369b11c51c209abc8a0ead9446eda61756804dae621239af04b88705801d6f9f"} Jul 24 18:43:29.484249 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.484186 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bfzgn" event={"ID":"d6d3da16-d2a0-4046-b297-759095781b40","Type":"ContainerStarted","Data":"6155fb37c834086257466784fdcf3c63bef00457a86c722026c7c08d47fedde8"} Jul 24 18:43:29.489388 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.489334 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" event={"ID":"ce6f309d-c9c3-478b-ba2e-d5680d409922","Type":"ContainerStarted","Data":"1f246cab58e61595f409399d69d17487bd3c5deb47652d7b374b000ed7f87ca4"} Jul 24 18:43:29.502788 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.502692 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dvnrx" event={"ID":"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f","Type":"ContainerStarted","Data":"263d6f9c78b38ed0974a7e6bf810fe423255a151a14e46023deb2f2db6804ee1"} Jul 24 18:43:29.516341 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.516309 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" event={"ID":"82e05024-15c2-4996-836d-3bf54836445c","Type":"ContainerStarted","Data":"77961d59be7d67e765e9bcb61d5653b869ae161f163301fd0301d1b727097a5a"} Jul 24 18:43:29.529849 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.529707 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" event={"ID":"635e0f85-09ee-4cd3-a549-4d5fb3db9b48","Type":"ContainerStarted","Data":"2d2a160268f6379d59ffaa90e9789f23c3fa3bb0961d060893e22142f9ff27b3"} Jul 24 18:43:29.532134 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.532105 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psph6" event={"ID":"14752100-7a25-4f88-abc5-c6903e11289e","Type":"ContainerStarted","Data":"2bf67707f9356fe3f05561b0a91516f12ae29a577c610d2dd67f0f303dcc71c9"} Jul 24 18:43:29.533723 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.533665 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pgx49" event={"ID":"7d43cff0-e130-45e6-8fbc-e212008b30a4","Type":"ContainerStarted","Data":"88b5a3c031f8e700483e3c67cd0cc5b2b34d3953a964a81c8199ec999db8b00f"} Jul 24 18:43:29.544851 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.544786 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s29nq" event={"ID":"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1","Type":"ContainerStarted","Data":"84b6a92f4cdd4b76768af703241480fd918ec782fbe646022c0f84b7d09f9c3a"} Jul 24 18:43:29.547627 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:29.547569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rlnzb" event={"ID":"fee3297a-eb1f-4807-b94b-a8b57a303292","Type":"ContainerStarted","Data":"6822a632e0839e77751c9be9f486e06d081493b8918681e85c781ecf2d2e45a4"} Jul 24 18:43:30.567513 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:30.567478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" event={"ID":"758f59bb62b76599c47fbe538d30ba69","Type":"ContainerStarted","Data":"25d0e0e7c44e4abb1c5aa37b3bf32077109f5578d146ae91aaf5ee28046056a6"} Jul 24 18:43:30.582969 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:30.582911 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-30.ec2.internal" podStartSLOduration=4.582891146 podStartE2EDuration="4.582891146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:43:29.493335545 +0000 UTC m=+4.569848107" watchObservedRunningTime="2026-07-24 18:43:30.582891146 +0000 UTC m=+5.659403705" Jul 24 18:43:30.991940 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:30.991901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:30.992190 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:30.992170 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:30.992281 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:30.992240 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs podName:3ed67684-a72a-4ec0-9a45-3871b53a3e72 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:34.992222258 +0000 UTC m=+10.068734810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs") pod "network-metrics-daemon-khpvw" (UID: "3ed67684-a72a-4ec0-9a45-3871b53a3e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:31.092573 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:31.092534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:31.092776 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:31.092713 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:31.092833 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:31.092802 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret podName:2a976a8d-b47d-4dbb-a8cd-f9a7170875a8 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:35.092782844 +0000 UTC m=+10.169295385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret") pod "global-pull-secret-syncer-5q8wz" (UID: "2a976a8d-b47d-4dbb-a8cd-f9a7170875a8") : object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:31.194638 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:31.193880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwq7\" (UniqueName: \"kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7\") pod \"network-check-target-x6l9k\" (UID: \"641810ec-5d18-41c3-990b-462e721aa175\") " pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:31.194638 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:31.194130 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 18:43:31.194638 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:31.194154 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 18:43:31.194638 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:31.194167 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5qwq7 for pod openshift-network-diagnostics/network-check-target-x6l9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:31.194638 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:31.194258 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7 podName:641810ec-5d18-41c3-990b-462e721aa175 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:35.19423977 +0000 UTC m=+10.270752320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qwq7" (UniqueName: "kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7") pod "network-check-target-x6l9k" (UID: "641810ec-5d18-41c3-990b-462e721aa175") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:31.414037 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:31.413138 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:31.414037 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:31.413279 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:31.414037 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:31.413652 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:31.414037 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:31.413732 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:31.414037 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:31.413812 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:31.414037 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:31.413884 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:33.413179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:33.413151 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:33.413673 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:33.413277 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:33.413673 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:33.413648 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:33.413828 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:33.413757 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:33.413828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:33.413810 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:33.413951 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:33.413881 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:35.028530 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:35.028490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:35.029009 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.028607 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:35.029009 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.028772 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs podName:3ed67684-a72a-4ec0-9a45-3871b53a3e72 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:43.028751588 +0000 UTC m=+18.105264125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs") pod "network-metrics-daemon-khpvw" (UID: "3ed67684-a72a-4ec0-9a45-3871b53a3e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:35.130262 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:35.129734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:35.130262 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.129876 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:35.130262 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.129935 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret podName:2a976a8d-b47d-4dbb-a8cd-f9a7170875a8 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:43.12991795 +0000 UTC m=+18.206430500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret") pod "global-pull-secret-syncer-5q8wz" (UID: "2a976a8d-b47d-4dbb-a8cd-f9a7170875a8") : object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:35.230547 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:35.230511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwq7\" (UniqueName: \"kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7\") pod \"network-check-target-x6l9k\" (UID: \"641810ec-5d18-41c3-990b-462e721aa175\") " pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:35.230735 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.230660 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 18:43:35.230735 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.230683 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 18:43:35.230735 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.230695 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5qwq7 for pod openshift-network-diagnostics/network-check-target-x6l9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:35.230928 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.230774 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7 podName:641810ec-5d18-41c3-990b-462e721aa175 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:43.230754996 +0000 UTC m=+18.307267547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qwq7" (UniqueName: "kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7") pod "network-check-target-x6l9k" (UID: "641810ec-5d18-41c3-990b-462e721aa175") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:35.425592 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:35.425553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:35.425799 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.425681 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:35.427433 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:35.427088 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:35.427433 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.427236 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:35.427433 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:35.427279 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:35.427433 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:35.427383 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:37.410868 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:37.410832 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:37.411269 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:37.410838 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:37.411269 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:37.410968 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:37.411269 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:37.410970 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:37.411269 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:37.411069 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:37.411269 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:37.411168 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:39.411394 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:39.411360 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:39.411849 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:39.411366 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:39.411849 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:39.411502 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:39.411849 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:39.411374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:39.411849 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:39.411567 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:39.411849 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:39.411635 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:41.411112 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:41.411074 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:41.411554 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:41.411074 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:41.411554 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:41.411074 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:41.411554 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:41.411217 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:41.411554 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:41.411334 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:41.411554 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:41.411415 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:43.091876 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:43.091845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:43.092251 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.091949 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:43.092251 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.091998 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs podName:3ed67684-a72a-4ec0-9a45-3871b53a3e72 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:59.091984122 +0000 UTC m=+34.168496657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs") pod "network-metrics-daemon-khpvw" (UID: "3ed67684-a72a-4ec0-9a45-3871b53a3e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:43.192267 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:43.192237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:43.192399 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.192387 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:43.192457 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.192446 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret podName:2a976a8d-b47d-4dbb-a8cd-f9a7170875a8 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:59.192430668 +0000 UTC m=+34.268943204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret") pod "global-pull-secret-syncer-5q8wz" (UID: "2a976a8d-b47d-4dbb-a8cd-f9a7170875a8") : object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:43.293529 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:43.293491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwq7\" (UniqueName: \"kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7\") pod \"network-check-target-x6l9k\" (UID: \"641810ec-5d18-41c3-990b-462e721aa175\") " pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:43.293722 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.293677 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 18:43:43.293722 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.293700 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 18:43:43.293722 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.293711 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5qwq7 for pod openshift-network-diagnostics/network-check-target-x6l9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:43.293912 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.293790 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7 podName:641810ec-5d18-41c3-990b-462e721aa175 nodeName:}" failed. No retries permitted until 2026-07-24 18:43:59.293774753 +0000 UTC m=+34.370287290 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qwq7" (UniqueName: "kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7") pod "network-check-target-x6l9k" (UID: "641810ec-5d18-41c3-990b-462e721aa175") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:43.410714 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:43.410676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:43.410714 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:43.410704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:43.410942 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.410800 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:43.410942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:43.410853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:43.411038 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.410960 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:43.411038 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:43.411024 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:45.410969 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:45.410918 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:45.411464 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:45.411061 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:45.411464 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:45.411438 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:45.411591 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:45.411511 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:45.411591 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:45.411575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:45.411687 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:45.411630 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:47.411412 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.410938 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:47.411412 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.410951 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:47.411412 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.411044 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:47.411412 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:47.411064 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:47.411412 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:47.411140 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:47.411412 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:47.411220 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:47.605561 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.605327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" event={"ID":"82e05024-15c2-4996-836d-3bf54836445c","Type":"ContainerStarted","Data":"64d64d1ad38beda6887218016625b3220142a898d000c9e0492483719ee19009"} Jul 24 18:43:47.607337 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.607315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" event={"ID":"635e0f85-09ee-4cd3-a549-4d5fb3db9b48","Type":"ContainerStarted","Data":"f1e1ec7ca9cec5b10f7991e92d4c131f544634c5f941532e392ecfafb2f5265c"} Jul 24 18:43:47.608612 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.608577 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psph6" event={"ID":"14752100-7a25-4f88-abc5-c6903e11289e","Type":"ContainerStarted","Data":"8e68a7c705f885e0d5860bfad1a76e8739ed5e1fd5876eb29b2797edc333191a"} Jul 24 18:43:47.610128 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.610107 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pgx49" event={"ID":"7d43cff0-e130-45e6-8fbc-e212008b30a4","Type":"ContainerStarted","Data":"63d986abd8f680763a9fc627865f136659910deaa6744ff5c06e93cc5d78998a"} Jul 24 18:43:47.611142 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.611124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s29nq" event={"ID":"e6e2711c-af0a-41eb-b4d0-d1d86509fdc1","Type":"ContainerStarted","Data":"791170fc4a37978acf424f7ee90d6a90584e2f1201a66da33573203c86437986"} Jul 24 18:43:47.612045 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.612027 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rlnzb" event={"ID":"fee3297a-eb1f-4807-b94b-a8b57a303292","Type":"ContainerStarted","Data":"2fc4f78436b79292232fb8b7ee1015c0f736151ab1df307839c031a4d6327468"} Jul 24 18:43:47.614404 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.613350 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bfzgn" event={"ID":"d6d3da16-d2a0-4046-b297-759095781b40","Type":"ContainerStarted","Data":"75df8aea602308a48a387add87dc5a8128ec2c64fb3f62ff9832f0a785d2fb90"} Jul 24 18:43:47.615500 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.615482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" event={"ID":"ce6f309d-c9c3-478b-ba2e-d5680d409922","Type":"ContainerStarted","Data":"d2e57be5ac274de25280cba02578f5ea33aa321578132ffd1d4c2a9a5d5b69fd"} Jul 24 18:43:47.640892 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.640847 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-30.ec2.internal" podStartSLOduration=21.640834407 podStartE2EDuration="21.640834407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:43:30.58399213 +0000 UTC m=+5.660504683" watchObservedRunningTime="2026-07-24 18:43:47.640834407 +0000 UTC m=+22.717346965" Jul 24 18:43:47.677576 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.677518 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cb6lh" podStartSLOduration=3.980724644 podStartE2EDuration="22.677501476s" totalImagesPullingTime="18.696776832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:43:47.661069511 +0000 UTC m=+22.737582070" watchObservedRunningTime="2026-07-24 18:43:47.677501476 +0000 UTC m=+22.754014090" Jul 24 18:43:47.677934 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.677905 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rlnzb" podStartSLOduration=12.259368703 podStartE2EDuration="22.677895864s" totalImagesPullingTime="10.418527161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:43:47.677238239 +0000 UTC m=+22.753750796" watchObservedRunningTime="2026-07-24 18:43:47.677895864 +0000 UTC m=+22.754408424" Jul 24 18:43:47.702011 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.701963 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bfzgn" podStartSLOduration=3.962492744 podStartE2EDuration="22.701947717s" totalImagesPullingTime="18.739454973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:43:47.701094514 +0000 UTC m=+22.777607071" watchObservedRunningTime="2026-07-24 18:43:47.701947717 +0000 UTC m=+22.778460275" Jul 24 18:43:47.732813 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.732725 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pgx49" podStartSLOduration=4.034934597 podStartE2EDuration="22.732703053s" totalImagesPullingTime="18.697768456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:43:47.732278935 +0000 UTC m=+22.808791552" watchObservedRunningTime="2026-07-24 18:43:47.732703053 +0000 UTC m=+22.809215612" Jul 24 18:43:47.748894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:47.748845 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s29nq" podStartSLOduration=4.035451649 podStartE2EDuration="22.748828515s" totalImagesPullingTime="18.713376866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:43:47.748335644 +0000 UTC m=+22.824848215" watchObservedRunningTime="2026-07-24 18:43:47.748828515 +0000 UTC m=+22.825341073" Jul 24 18:43:48.619984 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.619805 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 18:43:48.620794 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.620259 2575 generic.go:356] "Generic (PLEG): container finished" podID="635e0f85-09ee-4cd3-a549-4d5fb3db9b48" containerID="cf052f027085f6d8c2e85468424ccadd09948265de34bd0bbe846c5969cc4671" exitCode=1 Jul 24 18:43:48.620794 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.620325 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" event={"ID":"635e0f85-09ee-4cd3-a549-4d5fb3db9b48","Type":"ContainerStarted","Data":"ea453ebfb143a86dbff148bf8edb0f1e5e013afca8287e6a855fb0b6edc80dc7"} Jul 24 18:43:48.620794 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.620423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" event={"ID":"635e0f85-09ee-4cd3-a549-4d5fb3db9b48","Type":"ContainerStarted","Data":"c2236e08e2b4ec00055faf33aa69db572ad90730750e171927ce16ed031fa491"} Jul 24 18:43:48.620794 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.620459 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" event={"ID":"635e0f85-09ee-4cd3-a549-4d5fb3db9b48","Type":"ContainerStarted","Data":"59edaa92c90b4669fcc0c8014b89a1cb9e30a013a3ef4d40fb85d81bb92c007a"} Jul 24 18:43:48.620794 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.620487 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" event={"ID":"635e0f85-09ee-4cd3-a549-4d5fb3db9b48","Type":"ContainerStarted","Data":"0ac15b7974255843fc22108053f41b92414c860e59df40403ac7744cdfe3c082"} Jul 24 18:43:48.620794 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.620521 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" event={"ID":"635e0f85-09ee-4cd3-a549-4d5fb3db9b48","Type":"ContainerDied","Data":"cf052f027085f6d8c2e85468424ccadd09948265de34bd0bbe846c5969cc4671"} Jul 24 18:43:48.621595 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.621580 2575 generic.go:356] "Generic (PLEG): container finished" podID="14752100-7a25-4f88-abc5-c6903e11289e" containerID="8e68a7c705f885e0d5860bfad1a76e8739ed5e1fd5876eb29b2797edc333191a" exitCode=0 Jul 24 18:43:48.621661 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.621645 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psph6" event={"ID":"14752100-7a25-4f88-abc5-c6903e11289e","Type":"ContainerDied","Data":"8e68a7c705f885e0d5860bfad1a76e8739ed5e1fd5876eb29b2797edc333191a"} Jul 24 18:43:48.623114 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.623055 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dvnrx" event={"ID":"0484e1e0-0c45-4f9a-a2a8-9882a8dcb14f","Type":"ContainerStarted","Data":"1b6eec8df31e623111806e50b95380d8f0ebbb0b5b4b3e0f27f29f492c4ad8cd"} Jul 24 18:43:48.666970 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:48.666925 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dvnrx" podStartSLOduration=4.963215711 podStartE2EDuration="23.666906607s" totalImagesPullingTime="18.703690896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:43:48.666863261 +0000 UTC m=+23.743375818" watchObservedRunningTime="2026-07-24 18:43:48.666906607 +0000 UTC m=+23.743419166" Jul 24 18:43:49.089098 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:49.089071 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 18:43:49.407252 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:49.407098 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T18:43:49.089091737Z","UUID":"13206a73-db89-4afc-b65e-e74cc99e3bc4","Handler":null,"Name":"","Endpoint":""} Jul 24 18:43:49.408756 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:49.408707 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 18:43:49.408892 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:49.408774 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 18:43:49.411522 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:49.411115 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:49.411522 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:49.411212 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:49.411522 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:49.411438 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:49.411522 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:49.411494 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:49.411729 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:49.411615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:49.411792 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:49.411721 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:49.627337 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:49.627293 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" event={"ID":"82e05024-15c2-4996-836d-3bf54836445c","Type":"ContainerStarted","Data":"1e818da61f9b29636c4f5c8762d5e68c518751075cc3b4702ee9766ae95f7c61"} Jul 24 18:43:50.051787 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:50.051756 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:50.052428 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:50.052393 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:50.631561 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:50.631279 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" event={"ID":"82e05024-15c2-4996-836d-3bf54836445c","Type":"ContainerStarted","Data":"8e1385d421106832205c3286ce66ecba3794088625fea4eda980f9a2b3d9a527"} Jul 24 18:43:50.634176 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:50.634155 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 18:43:50.634480 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:50.634457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" event={"ID":"635e0f85-09ee-4cd3-a549-4d5fb3db9b48","Type":"ContainerStarted","Data":"0ec08906cc528027bd2a0b22a59f6d2acde290103b9d5bf4ba646a1ea215342f"} Jul 24 18:43:50.634665 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:50.634649 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:50.635166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:50.635146 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rlnzb" Jul 24 18:43:50.648818 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:50.648771 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zc245" podStartSLOduration=4.554007229 podStartE2EDuration="25.648756136s" totalImagesPullingTime="21.094748907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:43:50.648619633 +0000 UTC m=+25.725132202" watchObservedRunningTime="2026-07-24 18:43:50.648756136 +0000 UTC m=+25.725268690" Jul 24 18:43:51.411180 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:51.411145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:51.411416 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:51.411145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:51.411416 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:51.411278 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:51.411416 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:51.411145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:51.411416 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:51.411356 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:51.411623 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:51.411417 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:53.411040 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:53.411010 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:53.411040 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:53.411028 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:53.411828 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:53.411033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:53.411828 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:53.411141 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:53.411828 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:53.411219 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:53.411828 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:53.411289 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:53.646760 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:53.646486 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 18:43:53.647613 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:53.647028 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" event={"ID":"635e0f85-09ee-4cd3-a549-4d5fb3db9b48","Type":"ContainerStarted","Data":"c50dfda44c67b2ff7549bbcd5e5ac307b808df7f68ddebee5cc67f085055b33b"} Jul 24 18:43:53.647613 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:53.647538 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:53.647794 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:53.647620 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:53.647989 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:53.647840 2575 scope.go:117] "RemoveContainer" containerID="cf052f027085f6d8c2e85468424ccadd09948265de34bd0bbe846c5969cc4671" Jul 24 18:43:53.666833 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:53.666701 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:54.653038 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:54.652867 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 18:43:54.653483 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:54.653373 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" event={"ID":"635e0f85-09ee-4cd3-a549-4d5fb3db9b48","Type":"ContainerStarted","Data":"28dd5cd37af7df6f7cae08b805997b3f8bed9a12a36462d92ba0d4ff2b7c8aeb"} Jul 24 18:43:54.653549 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:54.653530 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:54.655405 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:54.655378 2575 generic.go:356] "Generic (PLEG): container finished" podID="14752100-7a25-4f88-abc5-c6903e11289e" containerID="ff56c99f48eff7281dbca84c0d576dab489cd65651f11087ca4bf91f4cf2b881" exitCode=0 Jul 24 18:43:54.655535 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:54.655423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psph6" event={"ID":"14752100-7a25-4f88-abc5-c6903e11289e","Type":"ContainerDied","Data":"ff56c99f48eff7281dbca84c0d576dab489cd65651f11087ca4bf91f4cf2b881"} Jul 24 18:43:54.670699 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:54.670681 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:43:54.688733 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:54.688692 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" podStartSLOduration=10.955207418 podStartE2EDuration="29.688679384s" totalImagesPullingTime="18.733471966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:43:54.688256575 +0000 UTC m=+29.764769132" watchObservedRunningTime="2026-07-24 18:43:54.688679384 +0000 UTC m=+29.765191941" Jul 24 18:43:55.411563 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.411523 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:55.411785 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.411653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:55.411785 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:55.411678 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:55.411785 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:55.411726 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:55.411962 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.411797 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:55.411962 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:55.411877 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:55.601614 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.601581 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-khpvw"] Jul 24 18:43:55.631281 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.631251 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5q8wz"] Jul 24 18:43:55.632005 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.631982 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x6l9k"] Jul 24 18:43:55.660020 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.659990 2575 generic.go:356] "Generic (PLEG): container finished" podID="14752100-7a25-4f88-abc5-c6903e11289e" containerID="9b622f8400c92cdc20d47b09c50aa3bec6a3960dced291d54100fc922ec62279" exitCode=0 Jul 24 18:43:55.660362 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.660052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:55.660362 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.660095 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psph6" event={"ID":"14752100-7a25-4f88-abc5-c6903e11289e","Type":"ContainerDied","Data":"9b622f8400c92cdc20d47b09c50aa3bec6a3960dced291d54100fc922ec62279"} Jul 24 18:43:55.660362 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:55.660284 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:55.660362 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.660312 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:55.660512 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:55.660403 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:55.661149 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:55.660661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:55.661149 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:55.660770 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:56.664989 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:56.664784 2575 generic.go:356] "Generic (PLEG): container finished" podID="14752100-7a25-4f88-abc5-c6903e11289e" containerID="029207ecdd8dc7a0a77fb225ad64b18451e57947116a4aa7fc8b5db4fc32de76" exitCode=0 Jul 24 18:43:56.665423 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:56.664862 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psph6" event={"ID":"14752100-7a25-4f88-abc5-c6903e11289e","Type":"ContainerDied","Data":"029207ecdd8dc7a0a77fb225ad64b18451e57947116a4aa7fc8b5db4fc32de76"} Jul 24 18:43:57.410940 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:57.410910 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:57.411163 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:57.410959 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:57.411163 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:57.410918 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:57.411163 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:57.411040 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:57.411316 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:57.411184 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:57.411316 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:57.411268 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:43:59.107233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:59.107197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:59.107947 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.107337 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:59.107947 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.107406 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs podName:3ed67684-a72a-4ec0-9a45-3871b53a3e72 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:31.107386862 +0000 UTC m=+66.183899418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs") pod "network-metrics-daemon-khpvw" (UID: "3ed67684-a72a-4ec0-9a45-3871b53a3e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 18:43:59.207998 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:59.207956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:59.208145 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.208125 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:59.208205 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.208195 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret podName:2a976a8d-b47d-4dbb-a8cd-f9a7170875a8 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:31.208180923 +0000 UTC m=+66.284693463 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret") pod "global-pull-secret-syncer-5q8wz" (UID: "2a976a8d-b47d-4dbb-a8cd-f9a7170875a8") : object "kube-system"/"original-pull-secret" not registered Jul 24 18:43:59.309241 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:59.309209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwq7\" (UniqueName: \"kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7\") pod \"network-check-target-x6l9k\" (UID: \"641810ec-5d18-41c3-990b-462e721aa175\") " pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:59.309397 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.309378 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 18:43:59.309449 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.309406 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 18:43:59.309449 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.309419 2575 projected.go:196] Error preparing data for projected volume kube-api-access-5qwq7 for pod openshift-network-diagnostics/network-check-target-x6l9k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:59.309529 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.309471 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7 podName:641810ec-5d18-41c3-990b-462e721aa175 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:31.30945661 +0000 UTC m=+66.385969149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qwq7" (UniqueName: "kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7") pod "network-check-target-x6l9k" (UID: "641810ec-5d18-41c3-990b-462e721aa175") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 18:43:59.413431 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:59.413403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:43:59.413592 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:59.413410 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:43:59.413592 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.413506 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-khpvw" podUID="3ed67684-a72a-4ec0-9a45-3871b53a3e72" Jul 24 18:43:59.413592 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:43:59.413410 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:43:59.413592 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.413583 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5q8wz" podUID="2a976a8d-b47d-4dbb-a8cd-f9a7170875a8" Jul 24 18:43:59.413817 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:43:59.413673 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x6l9k" podUID="641810ec-5d18-41c3-990b-462e721aa175" Jul 24 18:44:00.219052 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.219011 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-30.ec2.internal" event="NodeReady" Jul 24 18:44:00.219488 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.219171 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 18:44:00.253400 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.253297 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d954f5747-7kxdd"] Jul 24 18:44:00.292209 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.292171 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-54h5x"] Jul 24 18:44:00.292374 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.292309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.295051 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.295029 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 18:44:00.295199 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.295052 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 18:44:00.295260 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.295226 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4z6xk\"" Jul 24 18:44:00.295367 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.295350 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 18:44:00.301700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.301677 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 18:44:00.314347 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.314325 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d954f5747-7kxdd"] Jul 24 18:44:00.314435 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.314361 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b5hsw"] Jul 24 18:44:00.314491 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.314455 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.317175 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.317058 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 18:44:00.317175 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.317074 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 18:44:00.317175 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.317064 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fjp2z\"" Jul 24 18:44:00.337801 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.337776 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b5hsw"] Jul 24 18:44:00.337914 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.337808 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b5hsw" Jul 24 18:44:00.337914 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.337817 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-54h5x"] Jul 24 18:44:00.340607 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.340587 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 18:44:00.340712 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.340697 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-twnl6\"" Jul 24 18:44:00.341091 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.341073 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 18:44:00.341188 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.341144 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 18:44:00.418069 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-certificates\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.418069 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-bound-sa-token\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.418279 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-image-registry-private-configuration\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.418279 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418201 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.418279 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-installation-pull-secrets\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.418279 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418251 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7wln\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-kube-api-access-j7wln\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.418279 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fdcs\" (UniqueName: \"kubernetes.io/projected/249a8d0d-04b2-4720-a52d-37c37ff7a4ac-kube-api-access-7fdcs\") pod \"ingress-canary-b5hsw\" (UID: \"249a8d0d-04b2-4720-a52d-37c37ff7a4ac\") " pod="openshift-ingress-canary/ingress-canary-b5hsw" Jul 24 18:44:00.418460 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418332 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c9qs\" (UniqueName: \"kubernetes.io/projected/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-kube-api-access-2c9qs\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.418460 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418366 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/249a8d0d-04b2-4720-a52d-37c37ff7a4ac-cert\") pod \"ingress-canary-b5hsw\" (UID: \"249a8d0d-04b2-4720-a52d-37c37ff7a4ac\") " pod="openshift-ingress-canary/ingress-canary-b5hsw" Jul 24 18:44:00.418460 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-config-volume\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.418460 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.418460 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/425fa486-2440-4be2-b6fb-34f609ab97d5-ca-trust-extracted\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.418460 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-tmp-dir\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.418625 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.418478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-trusted-ca\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.519285 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-image-registry-private-configuration\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.519285 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519268 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-installation-pull-secrets\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wln\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-kube-api-access-j7wln\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fdcs\" (UniqueName: \"kubernetes.io/projected/249a8d0d-04b2-4720-a52d-37c37ff7a4ac-kube-api-access-7fdcs\") pod \"ingress-canary-b5hsw\" (UID: \"249a8d0d-04b2-4720-a52d-37c37ff7a4ac\") " pod="openshift-ingress-canary/ingress-canary-b5hsw" Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c9qs\" (UniqueName: \"kubernetes.io/projected/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-kube-api-access-2c9qs\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/249a8d0d-04b2-4720-a52d-37c37ff7a4ac-cert\") pod \"ingress-canary-b5hsw\" (UID: \"249a8d0d-04b2-4720-a52d-37c37ff7a4ac\") " pod="openshift-ingress-canary/ingress-canary-b5hsw" Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:00.519425 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:00.519445 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d954f5747-7kxdd: secret "image-registry-tls" not found Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519446 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-config-volume\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/425fa486-2440-4be2-b6fb-34f609ab97d5-ca-trust-extracted\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.519511 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:00.519517 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls podName:425fa486-2440-4be2-b6fb-34f609ab97d5 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:01.019495514 +0000 UTC m=+36.096008064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls") pod "image-registry-5d954f5747-7kxdd" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5") : secret "image-registry-tls" not found Jul 24 18:44:00.520067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-tmp-dir\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.520067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-trusted-ca\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.520067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-certificates\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.520067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-bound-sa-token\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.520067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.519842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/425fa486-2440-4be2-b6fb-34f609ab97d5-ca-trust-extracted\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.520067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.520052 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-tmp-dir\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.520404 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.520379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-config-volume\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.520497 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:00.520481 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 18:44:00.520551 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:00.520532 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls podName:0627a68a-b4bf-4eed-ae3d-6a3d4f97505e nodeName:}" failed. No retries permitted until 2026-07-24 18:44:01.020517018 +0000 UTC m=+36.097029556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls") pod "dns-default-54h5x" (UID: "0627a68a-b4bf-4eed-ae3d-6a3d4f97505e") : secret "dns-default-metrics-tls" not found Jul 24 18:44:00.520719 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.520698 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-certificates\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.521072 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.521049 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-trusted-ca\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.523963 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.523943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-image-registry-private-configuration\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.524033 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.523968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/249a8d0d-04b2-4720-a52d-37c37ff7a4ac-cert\") pod \"ingress-canary-b5hsw\" (UID: \"249a8d0d-04b2-4720-a52d-37c37ff7a4ac\") " pod="openshift-ingress-canary/ingress-canary-b5hsw" Jul 24 18:44:00.524033 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.523977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-installation-pull-secrets\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.529213 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.529185 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fdcs\" (UniqueName: \"kubernetes.io/projected/249a8d0d-04b2-4720-a52d-37c37ff7a4ac-kube-api-access-7fdcs\") pod \"ingress-canary-b5hsw\" (UID: \"249a8d0d-04b2-4720-a52d-37c37ff7a4ac\") " pod="openshift-ingress-canary/ingress-canary-b5hsw" Jul 24 18:44:00.530175 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.530134 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-bound-sa-token\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.530827 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.530808 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c9qs\" (UniqueName: \"kubernetes.io/projected/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-kube-api-access-2c9qs\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:00.532320 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.532296 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wln\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-kube-api-access-j7wln\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:00.648004 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:00.647962 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b5hsw" Jul 24 18:44:01.025117 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.025073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:01.025282 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.025156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:01.025282 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:01.025257 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 18:44:01.025282 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:01.025279 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d954f5747-7kxdd: secret "image-registry-tls" not found Jul 24 18:44:01.025376 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:01.025306 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 18:44:01.025376 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:01.025358 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls podName:425fa486-2440-4be2-b6fb-34f609ab97d5 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:02.025335949 +0000 UTC m=+37.101848511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls") pod "image-registry-5d954f5747-7kxdd" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5") : secret "image-registry-tls" not found Jul 24 18:44:01.025440 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:01.025380 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls podName:0627a68a-b4bf-4eed-ae3d-6a3d4f97505e nodeName:}" failed. No retries permitted until 2026-07-24 18:44:02.025370239 +0000 UTC m=+37.101882778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls") pod "dns-default-54h5x" (UID: "0627a68a-b4bf-4eed-ae3d-6a3d4f97505e") : secret "dns-default-metrics-tls" not found Jul 24 18:44:01.147878 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.147680 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b5hsw"] Jul 24 18:44:01.263674 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:01.263580 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod249a8d0d_04b2_4720_a52d_37c37ff7a4ac.slice/crio-c436a639ae06800a362b7110d322bb4b79737d8046736f6c3f10fd38a564a2ce WatchSource:0}: Error finding container c436a639ae06800a362b7110d322bb4b79737d8046736f6c3f10fd38a564a2ce: Status 404 returned error can't find the container with id c436a639ae06800a362b7110d322bb4b79737d8046736f6c3f10fd38a564a2ce Jul 24 18:44:01.410947 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.410925 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:44:01.410947 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.410938 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:44:01.411101 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.411080 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:44:01.413873 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.413854 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 18:44:01.413973 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.413904 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 18:44:01.413973 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.413927 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 18:44:01.414927 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.414908 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-96zrd\"" Jul 24 18:44:01.415070 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.415053 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6w7xn\"" Jul 24 18:44:01.415167 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.415085 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 18:44:01.680888 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.680841 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b5hsw" event={"ID":"249a8d0d-04b2-4720-a52d-37c37ff7a4ac","Type":"ContainerStarted","Data":"c436a639ae06800a362b7110d322bb4b79737d8046736f6c3f10fd38a564a2ce"} Jul 24 18:44:01.683322 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.683294 2575 generic.go:356] "Generic (PLEG): container finished" podID="14752100-7a25-4f88-abc5-c6903e11289e" containerID="27e742bcb771e02ee2e93594ad32b93575f518ae583be92c4a4b29cb6de72ed1" exitCode=0 Jul 24 18:44:01.683446 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:01.683339 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psph6" event={"ID":"14752100-7a25-4f88-abc5-c6903e11289e","Type":"ContainerDied","Data":"27e742bcb771e02ee2e93594ad32b93575f518ae583be92c4a4b29cb6de72ed1"} Jul 24 18:44:02.034482 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:02.034397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:02.034482 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:02.034463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:02.034683 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:02.034562 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 18:44:02.034683 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:02.034633 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls podName:0627a68a-b4bf-4eed-ae3d-6a3d4f97505e nodeName:}" failed. No retries permitted until 2026-07-24 18:44:04.034614157 +0000 UTC m=+39.111126696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls") pod "dns-default-54h5x" (UID: "0627a68a-b4bf-4eed-ae3d-6a3d4f97505e") : secret "dns-default-metrics-tls" not found Jul 24 18:44:02.034969 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:02.034567 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 18:44:02.035068 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:02.034973 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d954f5747-7kxdd: secret "image-registry-tls" not found Jul 24 18:44:02.035068 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:02.035022 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls podName:425fa486-2440-4be2-b6fb-34f609ab97d5 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:04.035005414 +0000 UTC m=+39.111517969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls") pod "image-registry-5d954f5747-7kxdd" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5") : secret "image-registry-tls" not found Jul 24 18:44:02.689260 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:02.689218 2575 generic.go:356] "Generic (PLEG): container finished" podID="14752100-7a25-4f88-abc5-c6903e11289e" containerID="0dc0ceaae188349d84ae7727ee35fb6106a88f4ac45a60b6fdd5f8883c827fe7" exitCode=0 Jul 24 18:44:02.689700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:02.689270 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psph6" event={"ID":"14752100-7a25-4f88-abc5-c6903e11289e","Type":"ContainerDied","Data":"0dc0ceaae188349d84ae7727ee35fb6106a88f4ac45a60b6fdd5f8883c827fe7"} Jul 24 18:44:03.693919 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:03.693885 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-psph6" event={"ID":"14752100-7a25-4f88-abc5-c6903e11289e","Type":"ContainerStarted","Data":"96aa55673297451dcaa0b3cdace413a7a59002b3f76886467b031d1bca734789"} Jul 24 18:44:03.715902 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:03.715859 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-psph6" podStartSLOduration=9.128382054 podStartE2EDuration="38.715846114s" totalImagesPullingTime="29.58746406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:03.71409955 +0000 UTC m=+38.790612108" watchObservedRunningTime="2026-07-24 18:44:03.715846114 +0000 UTC m=+38.792358671" Jul 24 18:44:03.860255 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:03.859972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:03.958903 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:03.958816 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:04.050520 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:04.050485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:04.050698 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:04.050546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:04.050698 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:04.050629 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 18:44:04.050698 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:04.050650 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d954f5747-7kxdd: secret "image-registry-tls" not found Jul 24 18:44:04.050698 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:04.050671 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 18:44:04.050869 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:04.050704 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls podName:425fa486-2440-4be2-b6fb-34f609ab97d5 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:08.050688353 +0000 UTC m=+43.127200923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls") pod "image-registry-5d954f5747-7kxdd" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5") : secret "image-registry-tls" not found Jul 24 18:44:04.050869 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:04.050723 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls podName:0627a68a-b4bf-4eed-ae3d-6a3d4f97505e nodeName:}" failed. No retries permitted until 2026-07-24 18:44:08.050710873 +0000 UTC m=+43.127223409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls") pod "dns-default-54h5x" (UID: "0627a68a-b4bf-4eed-ae3d-6a3d4f97505e") : secret "dns-default-metrics-tls" not found Jul 24 18:44:04.697421 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:04.697385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b5hsw" event={"ID":"249a8d0d-04b2-4720-a52d-37c37ff7a4ac","Type":"ContainerStarted","Data":"ef71c89710005fb1ca534c2cb0a966762001dac5b95ccc365d16bbb3668661de"} Jul 24 18:44:04.718394 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:04.718344 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b5hsw" podStartSLOduration=2.261033818 podStartE2EDuration="4.718329083s" totalImagesPullingTime="2.457295265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:04.717328422 +0000 UTC m=+39.793841004" watchObservedRunningTime="2026-07-24 18:44:04.718329083 +0000 UTC m=+39.794841653" Jul 24 18:44:05.329153 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:05.329114 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:05.404570 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:05.404515 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:05.503862 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:05.503821 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:05.606056 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:05.606015 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:05.706002 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:05.705925 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:06.286114 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:06.286086 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:06.337202 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:06.337160 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:06.688450 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:06.688413 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:06.754016 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:06.752928 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:08.079806 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:08.079762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:08.080259 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:08.079849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:08.080259 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:08.079924 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 18:44:08.080259 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:08.079988 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 18:44:08.080259 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:08.079996 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls podName:0627a68a-b4bf-4eed-ae3d-6a3d4f97505e nodeName:}" failed. No retries permitted until 2026-07-24 18:44:16.079978762 +0000 UTC m=+51.156491326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls") pod "dns-default-54h5x" (UID: "0627a68a-b4bf-4eed-ae3d-6a3d4f97505e") : secret "dns-default-metrics-tls" not found Jul 24 18:44:08.080259 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:08.080002 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d954f5747-7kxdd: secret "image-registry-tls" not found Jul 24 18:44:08.080259 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:08.080051 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls podName:425fa486-2440-4be2-b6fb-34f609ab97d5 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:16.080035413 +0000 UTC m=+51.156547949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls") pod "image-registry-5d954f5747-7kxdd" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5") : secret "image-registry-tls" not found Jul 24 18:44:09.386066 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.386035 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:09.924090 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.924061 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc"] Jul 24 18:44:09.959033 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.958985 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc"] Jul 24 18:44:09.959185 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.959044 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr"] Jul 24 18:44:09.959185 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.959091 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc" Jul 24 18:44:09.962001 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.961970 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 18:44:09.962154 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.962054 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kzk2v\"" Jul 24 18:44:09.962154 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.962077 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 18:44:09.972460 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.972440 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b"] Jul 24 18:44:09.972566 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.972549 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr" Jul 24 18:44:09.975062 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.975046 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-z64qz\"" Jul 24 18:44:09.992581 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.992559 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-6cl5p"] Jul 24 18:44:09.992699 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.992673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:09.995634 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.995611 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 18:44:09.995634 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.995634 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 18:44:09.995861 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.995659 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 18:44:09.995861 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.995696 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-s9thx\"" Jul 24 18:44:09.995861 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:09.995715 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 18:44:10.016579 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.016557 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-p2c9m"] Jul 24 18:44:10.016755 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.016723 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.019633 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.019614 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 18:44:10.019724 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.019617 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 18:44:10.019900 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.019881 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 18:44:10.020019 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.019999 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jgplj\"" Jul 24 18:44:10.020325 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.020308 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 18:44:10.028331 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.028312 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 18:44:10.038699 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.038680 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr"] Jul 24 18:44:10.038802 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.038710 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-6cl5p"] Jul 24 18:44:10.038802 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.038723 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b"] Jul 24 18:44:10.038802 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.038760 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-p2c9m"] Jul 24 18:44:10.038917 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.038800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.041789 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.041766 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 18:44:10.041912 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.041805 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 18:44:10.041912 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.041831 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 18:44:10.041912 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.041884 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 18:44:10.041912 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.041911 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9dblc\"" Jul 24 18:44:10.046939 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.046922 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 18:44:10.057698 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.057676 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r"] Jul 24 18:44:10.077040 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.077017 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z"] Jul 24 18:44:10.077174 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.077154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:10.080194 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.080172 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 18:44:10.080326 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.080173 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 18:44:10.080326 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.080218 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 18:44:10.081195 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.081169 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 18:44:10.081195 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.081175 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x7xg2\"" Jul 24 18:44:10.095954 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.095928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pnb\" (UniqueName: \"kubernetes.io/projected/9ca248b3-674e-4ee9-969a-7ad2f903b1a0-kube-api-access-n7pnb\") pod \"volume-data-source-validator-549c6d54cf-bpcrc\" (UID: \"9ca248b3-674e-4ee9-969a-7ad2f903b1a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc" Jul 24 18:44:10.095954 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.095956 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be74898d-3d60-437f-9c90-5408b5fbb3a3-config\") pod \"service-ca-operator-5d6c75f84f-mww9b\" (UID: \"be74898d-3d60-437f-9c90-5408b5fbb3a3\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:10.096110 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.095975 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be74898d-3d60-437f-9c90-5408b5fbb3a3-serving-cert\") pod \"service-ca-operator-5d6c75f84f-mww9b\" (UID: \"be74898d-3d60-437f-9c90-5408b5fbb3a3\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:10.096110 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.096038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk4xl\" (UniqueName: \"kubernetes.io/projected/be74898d-3d60-437f-9c90-5408b5fbb3a3-kube-api-access-sk4xl\") pod \"service-ca-operator-5d6c75f84f-mww9b\" (UID: \"be74898d-3d60-437f-9c90-5408b5fbb3a3\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:10.096201 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.096132 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5wm\" (UniqueName: \"kubernetes.io/projected/0308bd2a-3da9-4c65-aab6-26176a2aab3f-kube-api-access-gg5wm\") pod \"network-check-source-7d88cb55d8-xvxgr\" (UID: \"0308bd2a-3da9-4c65-aab6-26176a2aab3f\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr" Jul 24 18:44:10.098721 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.098704 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4"] Jul 24 18:44:10.098838 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.098823 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.101518 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.101497 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 18:44:10.101633 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.101604 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-z7n25\"" Jul 24 18:44:10.101758 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.101728 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 18:44:10.101852 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.101777 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 18:44:10.101852 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.101803 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 18:44:10.118874 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.118854 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6f4d68c8f7-vqjh4"] Jul 24 18:44:10.118979 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.118966 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:10.121592 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.121567 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 18:44:10.121706 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.121585 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5c5v5\"" Jul 24 18:44:10.121706 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.121567 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 18:44:10.121815 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.121706 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 18:44:10.140961 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.140941 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r"] Jul 24 18:44:10.141075 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.140972 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z"] Jul 24 18:44:10.141075 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.140989 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4"] Jul 24 18:44:10.141075 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.141013 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f4d68c8f7-vqjh4"] Jul 24 18:44:10.141075 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.141071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.143534 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.143515 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 18:44:10.143909 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.143893 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 18:44:10.143993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.143910 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zh4sc\"" Jul 24 18:44:10.143993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.143966 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 18:44:10.144136 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.144121 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 18:44:10.144178 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.144137 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 18:44:10.144357 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.144343 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 18:44:10.196522 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5fc61054-0130-44ec-a896-11d6124b50d6-snapshots\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.196522 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc61054-0130-44ec-a896-11d6124b50d6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.196522 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4abb5614-6d9a-42f6-a655-cfbc38b39382-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z\" (UID: \"4abb5614-6d9a-42f6-a655-cfbc38b39382\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.196789 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7knvk\" (UniqueName: \"kubernetes.io/projected/c683dae8-6f44-455b-bcd6-d81c0834ad39-kube-api-access-7knvk\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.196789 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc61054-0130-44ec-a896-11d6124b50d6-tmp\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.196789 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abb5614-6d9a-42f6-a655-cfbc38b39382-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z\" (UID: \"4abb5614-6d9a-42f6-a655-cfbc38b39382\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.196789 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:10.196972 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be74898d-3d60-437f-9c90-5408b5fbb3a3-serving-cert\") pod \"service-ca-operator-5d6c75f84f-mww9b\" (UID: \"be74898d-3d60-437f-9c90-5408b5fbb3a3\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:10.196972 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196840 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9ca07cf2-39b7-4962-ad44-fb73e63c4005-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:10.196972 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftn9w\" (UniqueName: \"kubernetes.io/projected/9ca07cf2-39b7-4962-ad44-fb73e63c4005-kube-api-access-ftn9w\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:10.196972 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5nrz\" (UniqueName: \"kubernetes.io/projected/5fc61054-0130-44ec-a896-11d6124b50d6-kube-api-access-w5nrz\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.197151 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.196986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c683dae8-6f44-455b-bcd6-d81c0834ad39-serving-cert\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.197151 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.197027 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c683dae8-6f44-455b-bcd6-d81c0834ad39-config\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.197151 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.197052 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk4xl\" (UniqueName: \"kubernetes.io/projected/be74898d-3d60-437f-9c90-5408b5fbb3a3-kube-api-access-sk4xl\") pod \"service-ca-operator-5d6c75f84f-mww9b\" (UID: \"be74898d-3d60-437f-9c90-5408b5fbb3a3\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:10.197151 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.197084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt29k\" (UniqueName: \"kubernetes.io/projected/4abb5614-6d9a-42f6-a655-cfbc38b39382-kube-api-access-mt29k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z\" (UID: \"4abb5614-6d9a-42f6-a655-cfbc38b39382\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.197151 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.197110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be74898d-3d60-437f-9c90-5408b5fbb3a3-config\") pod \"service-ca-operator-5d6c75f84f-mww9b\" (UID: \"be74898d-3d60-437f-9c90-5408b5fbb3a3\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:10.197151 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.197138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc61054-0130-44ec-a896-11d6124b50d6-service-ca-bundle\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.197379 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.197156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5wm\" (UniqueName: \"kubernetes.io/projected/0308bd2a-3da9-4c65-aab6-26176a2aab3f-kube-api-access-gg5wm\") pod \"network-check-source-7d88cb55d8-xvxgr\" (UID: \"0308bd2a-3da9-4c65-aab6-26176a2aab3f\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr" Jul 24 18:44:10.198166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.197170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c683dae8-6f44-455b-bcd6-d81c0834ad39-trusted-ca\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.198166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.197594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pnb\" (UniqueName: \"kubernetes.io/projected/9ca248b3-674e-4ee9-969a-7ad2f903b1a0-kube-api-access-n7pnb\") pod \"volume-data-source-validator-549c6d54cf-bpcrc\" (UID: \"9ca248b3-674e-4ee9-969a-7ad2f903b1a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc" Jul 24 18:44:10.198166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.197641 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc61054-0130-44ec-a896-11d6124b50d6-serving-cert\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.198356 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.198259 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be74898d-3d60-437f-9c90-5408b5fbb3a3-config\") pod \"service-ca-operator-5d6c75f84f-mww9b\" (UID: \"be74898d-3d60-437f-9c90-5408b5fbb3a3\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:10.202421 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.202392 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/be74898d-3d60-437f-9c90-5408b5fbb3a3-serving-cert\") pod \"service-ca-operator-5d6c75f84f-mww9b\" (UID: \"be74898d-3d60-437f-9c90-5408b5fbb3a3\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:10.206244 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.206218 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pnb\" (UniqueName: \"kubernetes.io/projected/9ca248b3-674e-4ee9-969a-7ad2f903b1a0-kube-api-access-n7pnb\") pod \"volume-data-source-validator-549c6d54cf-bpcrc\" (UID: \"9ca248b3-674e-4ee9-969a-7ad2f903b1a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc" Jul 24 18:44:10.206409 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.206391 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk4xl\" (UniqueName: \"kubernetes.io/projected/be74898d-3d60-437f-9c90-5408b5fbb3a3-kube-api-access-sk4xl\") pod \"service-ca-operator-5d6c75f84f-mww9b\" (UID: \"be74898d-3d60-437f-9c90-5408b5fbb3a3\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:10.206861 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.206842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5wm\" (UniqueName: \"kubernetes.io/projected/0308bd2a-3da9-4c65-aab6-26176a2aab3f-kube-api-access-gg5wm\") pod \"network-check-source-7d88cb55d8-xvxgr\" (UID: \"0308bd2a-3da9-4c65-aab6-26176a2aab3f\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr" Jul 24 18:44:10.219725 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.219696 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:10.268613 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.268587 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc" Jul 24 18:44:10.280877 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.280853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr" Jul 24 18:44:10.299007 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.298989 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftn9w\" (UniqueName: \"kubernetes.io/projected/9ca07cf2-39b7-4962-ad44-fb73e63c4005-kube-api-access-ftn9w\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:10.299070 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5nrz\" (UniqueName: \"kubernetes.io/projected/5fc61054-0130-44ec-a896-11d6124b50d6-kube-api-access-w5nrz\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.299070 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c683dae8-6f44-455b-bcd6-d81c0834ad39-serving-cert\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.299144 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.299144 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c683dae8-6f44-455b-bcd6-d81c0834ad39-config\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.299239 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.299387 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt29k\" (UniqueName: \"kubernetes.io/projected/4abb5614-6d9a-42f6-a655-cfbc38b39382-kube-api-access-mt29k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z\" (UID: \"4abb5614-6d9a-42f6-a655-cfbc38b39382\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.299453 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc61054-0130-44ec-a896-11d6124b50d6-service-ca-bundle\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.299511 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c683dae8-6f44-455b-bcd6-d81c0834ad39-trusted-ca\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.299511 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz67d\" (UniqueName: \"kubernetes.io/projected/31882160-593a-41c1-8b4a-9025178b589f-kube-api-access-tz67d\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:10.299610 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc61054-0130-44ec-a896-11d6124b50d6-serving-cert\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.299610 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:10.299702 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5fc61054-0130-44ec-a896-11d6124b50d6-snapshots\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.299702 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc61054-0130-44ec-a896-11d6124b50d6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.299702 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4abb5614-6d9a-42f6-a655-cfbc38b39382-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z\" (UID: \"4abb5614-6d9a-42f6-a655-cfbc38b39382\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.299856 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7knvk\" (UniqueName: \"kubernetes.io/projected/c683dae8-6f44-455b-bcd6-d81c0834ad39-kube-api-access-7knvk\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.299856 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299733 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-default-certificate\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.299856 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc61054-0130-44ec-a896-11d6124b50d6-tmp\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.299856 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abb5614-6d9a-42f6-a655-cfbc38b39382-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z\" (UID: \"4abb5614-6d9a-42f6-a655-cfbc38b39382\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.300064 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299858 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:10.300064 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frzjv\" (UniqueName: \"kubernetes.io/projected/ecfae2bd-f425-407e-b825-d5505cd538ef-kube-api-access-frzjv\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.300064 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299927 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-stats-auth\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.300064 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9ca07cf2-39b7-4962-ad44-fb73e63c4005-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:10.300064 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.299987 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c683dae8-6f44-455b-bcd6-d81c0834ad39-config\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.300276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.300220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc61054-0130-44ec-a896-11d6124b50d6-service-ca-bundle\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.300565 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.300533 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c683dae8-6f44-455b-bcd6-d81c0834ad39-trusted-ca\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.300565 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.300546 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5fc61054-0130-44ec-a896-11d6124b50d6-snapshots\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.300813 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.300793 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:10.300888 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.300857 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9ca07cf2-39b7-4962-ad44-fb73e63c4005-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:10.301281 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.301140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:10.301281 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.301224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc61054-0130-44ec-a896-11d6124b50d6-tmp\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.301396 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.301336 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4abb5614-6d9a-42f6-a655-cfbc38b39382-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z\" (UID: \"4abb5614-6d9a-42f6-a655-cfbc38b39382\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.301458 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.301393 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls podName:9ca07cf2-39b7-4962-ad44-fb73e63c4005 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:10.801372029 +0000 UTC m=+45.877884578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-qtf2r" (UID: "9ca07cf2-39b7-4962-ad44-fb73e63c4005") : secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:10.301458 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.301426 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" Jul 24 18:44:10.301657 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.301639 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc61054-0130-44ec-a896-11d6124b50d6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.303473 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.303450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4abb5614-6d9a-42f6-a655-cfbc38b39382-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z\" (UID: \"4abb5614-6d9a-42f6-a655-cfbc38b39382\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.303590 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.303573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5fc61054-0130-44ec-a896-11d6124b50d6-serving-cert\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.304108 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.304087 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c683dae8-6f44-455b-bcd6-d81c0834ad39-serving-cert\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.309220 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.308381 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftn9w\" (UniqueName: \"kubernetes.io/projected/9ca07cf2-39b7-4962-ad44-fb73e63c4005-kube-api-access-ftn9w\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:10.310097 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.310039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5nrz\" (UniqueName: \"kubernetes.io/projected/5fc61054-0130-44ec-a896-11d6124b50d6-kube-api-access-w5nrz\") pod \"insights-operator-5885fdb759-6cl5p\" (UID: \"5fc61054-0130-44ec-a896-11d6124b50d6\") " pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.310312 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.310295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7knvk\" (UniqueName: \"kubernetes.io/projected/c683dae8-6f44-455b-bcd6-d81c0834ad39-kube-api-access-7knvk\") pod \"console-operator-584dcb6b54-p2c9m\" (UID: \"c683dae8-6f44-455b-bcd6-d81c0834ad39\") " pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.313562 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.313540 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt29k\" (UniqueName: \"kubernetes.io/projected/4abb5614-6d9a-42f6-a655-cfbc38b39382-kube-api-access-mt29k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z\" (UID: \"4abb5614-6d9a-42f6-a655-cfbc38b39382\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.327188 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.327088 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-6cl5p" Jul 24 18:44:10.347947 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.347917 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.406117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-default-certificate\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.406198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frzjv\" (UniqueName: \"kubernetes.io/projected/ecfae2bd-f425-407e-b825-d5505cd538ef-kube-api-access-frzjv\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.406227 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-stats-auth\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.406292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.406319 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.406376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz67d\" (UniqueName: \"kubernetes.io/projected/31882160-593a-41c1-8b4a-9025178b589f-kube-api-access-tz67d\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.406410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.406557 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.406621 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls podName:31882160-593a-41c1-8b4a-9025178b589f nodeName:}" failed. No retries permitted until 2026-07-24 18:44:10.906601947 +0000 UTC m=+45.983114499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls") pod "cluster-samples-operator-6748489656-g8rc4" (UID: "31882160-593a-41c1-8b4a-9025178b589f") : secret "samples-operator-tls" not found Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.407348 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.407414 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:10.907397093 +0000 UTC m=+45.983909646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : secret "router-metrics-certs-default" not found Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.408217 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:10.908201805 +0000 UTC m=+45.984714357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : configmap references non-existent config key: service-ca.crt Jul 24 18:44:10.408474 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.408288 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" Jul 24 18:44:10.419298 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.415324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-default-certificate\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.433020 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.431366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frzjv\" (UniqueName: \"kubernetes.io/projected/ecfae2bd-f425-407e-b825-d5505cd538ef-kube-api-access-frzjv\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.433020 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.432024 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz67d\" (UniqueName: \"kubernetes.io/projected/31882160-593a-41c1-8b4a-9025178b589f-kube-api-access-tz67d\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:10.433020 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.432405 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-stats-auth\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.480178 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.478782 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc"] Jul 24 18:44:10.500386 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.500309 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr"] Jul 24 18:44:10.539813 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.539723 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b"] Jul 24 18:44:10.543401 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:10.543350 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe74898d_3d60_437f_9c90_5408b5fbb3a3.slice/crio-a3a5f6f433de1ef346da9e8c0aa5ce3dc18be1474a8944e9b4344edd7e7a9e00 WatchSource:0}: Error finding container a3a5f6f433de1ef346da9e8c0aa5ce3dc18be1474a8944e9b4344edd7e7a9e00: Status 404 returned error can't find the container with id a3a5f6f433de1ef346da9e8c0aa5ce3dc18be1474a8944e9b4344edd7e7a9e00 Jul 24 18:44:10.553452 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.553428 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-6cl5p"] Jul 24 18:44:10.556326 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:10.556298 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc61054_0130_44ec_a896_11d6124b50d6.slice/crio-4a914c3bd7caf950bb7b096de805efba1318b84da11ec330e2587492309acda1 WatchSource:0}: Error finding container 4a914c3bd7caf950bb7b096de805efba1318b84da11ec330e2587492309acda1: Status 404 returned error can't find the container with id 4a914c3bd7caf950bb7b096de805efba1318b84da11ec330e2587492309acda1 Jul 24 18:44:10.574338 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.574312 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-p2c9m"] Jul 24 18:44:10.576843 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:10.576815 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc683dae8_6f44_455b_bcd6_d81c0834ad39.slice/crio-e14ae159399025a8b0303a38c94680158cf94b8b82aa1b641e6160902e497b00 WatchSource:0}: Error finding container e14ae159399025a8b0303a38c94680158cf94b8b82aa1b641e6160902e497b00: Status 404 returned error can't find the container with id e14ae159399025a8b0303a38c94680158cf94b8b82aa1b641e6160902e497b00 Jul 24 18:44:10.618454 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.618426 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z"] Jul 24 18:44:10.621670 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:10.621642 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abb5614_6d9a_42f6_a655_cfbc38b39382.slice/crio-ea7eac1437f1af1eb7b7414ae494aec871a382a393815fc6a88135c636f65b75 WatchSource:0}: Error finding container ea7eac1437f1af1eb7b7414ae494aec871a382a393815fc6a88135c636f65b75: Status 404 returned error can't find the container with id ea7eac1437f1af1eb7b7414ae494aec871a382a393815fc6a88135c636f65b75 Jul 24 18:44:10.712030 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.711931 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" event={"ID":"be74898d-3d60-437f-9c90-5408b5fbb3a3","Type":"ContainerStarted","Data":"a3a5f6f433de1ef346da9e8c0aa5ce3dc18be1474a8944e9b4344edd7e7a9e00"} Jul 24 18:44:10.713057 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.713031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-6cl5p" event={"ID":"5fc61054-0130-44ec-a896-11d6124b50d6","Type":"ContainerStarted","Data":"4a914c3bd7caf950bb7b096de805efba1318b84da11ec330e2587492309acda1"} Jul 24 18:44:10.714104 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.714083 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc" event={"ID":"9ca248b3-674e-4ee9-969a-7ad2f903b1a0","Type":"ContainerStarted","Data":"4c816d565e6a5fe4171d5555f42502db8f91d3a9a828b42d6db90fc1dedb414f"} Jul 24 18:44:10.715132 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.715112 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr" event={"ID":"0308bd2a-3da9-4c65-aab6-26176a2aab3f","Type":"ContainerStarted","Data":"d61f85c9ee477ef071fc1c4609f22ab45d7c935b09bd536ab776f430a8c49b05"} Jul 24 18:44:10.716151 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.716131 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" event={"ID":"4abb5614-6d9a-42f6-a655-cfbc38b39382","Type":"ContainerStarted","Data":"ea7eac1437f1af1eb7b7414ae494aec871a382a393815fc6a88135c636f65b75"} Jul 24 18:44:10.717132 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.717111 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" event={"ID":"c683dae8-6f44-455b-bcd6-d81c0834ad39","Type":"ContainerStarted","Data":"e14ae159399025a8b0303a38c94680158cf94b8b82aa1b641e6160902e497b00"} Jul 24 18:44:10.809593 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.809556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:10.809783 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.809695 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:10.809837 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.809787 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls podName:9ca07cf2-39b7-4962-ad44-fb73e63c4005 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:11.809762856 +0000 UTC m=+46.886275412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-qtf2r" (UID: "9ca07cf2-39b7-4962-ad44-fb73e63c4005") : secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:10.910274 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.910238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.910274 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.910279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:10.910494 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:10.910353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:10.910494 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.910392 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 18:44:10.910494 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.910463 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:11.910442157 +0000 UTC m=+46.986954713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : secret "router-metrics-certs-default" not found Jul 24 18:44:10.910494 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.910477 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 18:44:10.910654 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.910498 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:11.910477491 +0000 UTC m=+46.986990044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : configmap references non-existent config key: service-ca.crt Jul 24 18:44:10.910654 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:10.910528 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls podName:31882160-593a-41c1-8b4a-9025178b589f nodeName:}" failed. No retries permitted until 2026-07-24 18:44:11.910517089 +0000 UTC m=+46.987029628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls") pod "cluster-samples-operator-6748489656-g8rc4" (UID: "31882160-593a-41c1-8b4a-9025178b589f") : secret "samples-operator-tls" not found Jul 24 18:44:11.177469 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:11.177431 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:11.246484 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:11.246445 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:11.820727 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:11.820676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:11.821194 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:11.820956 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:11.821194 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:11.821025 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls podName:9ca07cf2-39b7-4962-ad44-fb73e63c4005 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:13.821005935 +0000 UTC m=+48.897518490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-qtf2r" (UID: "9ca07cf2-39b7-4962-ad44-fb73e63c4005") : secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:11.921716 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:11.921678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:11.921911 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:11.921735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:11.921911 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:11.921892 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 18:44:11.922044 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:11.921900 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:13.921881354 +0000 UTC m=+48.998393894 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : configmap references non-existent config key: service-ca.crt Jul 24 18:44:11.922044 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:11.921978 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:13.921960246 +0000 UTC m=+48.998472786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : secret "router-metrics-certs-default" not found Jul 24 18:44:11.922163 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:11.922039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:11.922273 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:11.922256 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 18:44:11.922335 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:11.922302 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls podName:31882160-593a-41c1-8b4a-9025178b589f nodeName:}" failed. No retries permitted until 2026-07-24 18:44:13.922289972 +0000 UTC m=+48.998802508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls") pod "cluster-samples-operator-6748489656-g8rc4" (UID: "31882160-593a-41c1-8b4a-9025178b589f") : secret "samples-operator-tls" not found Jul 24 18:44:12.727004 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:12.726203 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc" event={"ID":"9ca248b3-674e-4ee9-969a-7ad2f903b1a0","Type":"ContainerStarted","Data":"9ecccf0dfc4e73fdb521b30ba8c1e709bcb4b5dfc7d0329b5402365483dc6d1c"} Jul 24 18:44:12.744281 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:12.743816 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-bpcrc" podStartSLOduration=1.717743908 podStartE2EDuration="3.743793035s" totalImagesPullingTime="2.026049127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:12.742669616 +0000 UTC m=+47.819182186" watchObservedRunningTime="2026-07-24 18:44:12.743793035 +0000 UTC m=+47.820305593" Jul 24 18:44:12.837473 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:12.837333 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:12.900799 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:12.900642 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:13.846459 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:13.846410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:13.846892 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:13.846872 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:13.846960 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:13.846950 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls podName:9ca07cf2-39b7-4962-ad44-fb73e63c4005 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:17.8469312 +0000 UTC m=+52.923443767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-qtf2r" (UID: "9ca07cf2-39b7-4962-ad44-fb73e63c4005") : secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:13.947819 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:13.947783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:13.947994 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:13.947904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:13.947994 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:13.947936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:13.947994 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:13.947962 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 18:44:13.948142 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:13.948044 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls podName:31882160-593a-41c1-8b4a-9025178b589f nodeName:}" failed. No retries permitted until 2026-07-24 18:44:17.948027476 +0000 UTC m=+53.024540015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls") pod "cluster-samples-operator-6748489656-g8rc4" (UID: "31882160-593a-41c1-8b4a-9025178b589f") : secret "samples-operator-tls" not found Jul 24 18:44:13.948142 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:13.948056 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 18:44:13.948142 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:13.948087 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:17.948071934 +0000 UTC m=+53.024584470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : configmap references non-existent config key: service-ca.crt Jul 24 18:44:13.948142 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:13.948111 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:17.948099812 +0000 UTC m=+53.024612349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : secret "router-metrics-certs-default" not found Jul 24 18:44:16.170091 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:16.169719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:16.170091 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:16.169856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:16.170091 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:16.169880 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 18:44:16.170091 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:16.169911 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d954f5747-7kxdd: secret "image-registry-tls" not found Jul 24 18:44:16.170091 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:16.169980 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls podName:425fa486-2440-4be2-b6fb-34f609ab97d5 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:32.169959077 +0000 UTC m=+67.246471666 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls") pod "image-registry-5d954f5747-7kxdd" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5") : secret "image-registry-tls" not found Jul 24 18:44:16.170091 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:16.170018 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 18:44:16.170091 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:16.170064 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls podName:0627a68a-b4bf-4eed-ae3d-6a3d4f97505e nodeName:}" failed. No retries permitted until 2026-07-24 18:44:32.170052192 +0000 UTC m=+67.246564728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls") pod "dns-default-54h5x" (UID: "0627a68a-b4bf-4eed-ae3d-6a3d4f97505e") : secret "dns-default-metrics-tls" not found Jul 24 18:44:17.746763 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.746706 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-6cl5p" event={"ID":"5fc61054-0130-44ec-a896-11d6124b50d6","Type":"ContainerStarted","Data":"03f7e231f928f2a9b2ae79d77a623fb2901fadfc56844e707e08493cda0069b5"} Jul 24 18:44:17.748225 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.748195 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr" event={"ID":"0308bd2a-3da9-4c65-aab6-26176a2aab3f","Type":"ContainerStarted","Data":"be8dc3c02c7e86c06c030c8fb66a7ddac26c197e187be43337d67cc2a62b0905"} Jul 24 18:44:17.749757 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.749716 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" event={"ID":"4abb5614-6d9a-42f6-a655-cfbc38b39382","Type":"ContainerStarted","Data":"cef8659e4c56ebd36dbfed034854e00fd99e0db772e86a0b35d40ac53ef7b77c"} Jul 24 18:44:17.751413 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.751393 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/0.log" Jul 24 18:44:17.751495 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.751431 2575 generic.go:356] "Generic (PLEG): container finished" podID="c683dae8-6f44-455b-bcd6-d81c0834ad39" containerID="00866783f35039a445c9162658106a7ac6f3aa6ceda7fb64de7277bafb3e1849" exitCode=255 Jul 24 18:44:17.751570 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.751516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" event={"ID":"c683dae8-6f44-455b-bcd6-d81c0834ad39","Type":"ContainerDied","Data":"00866783f35039a445c9162658106a7ac6f3aa6ceda7fb64de7277bafb3e1849"} Jul 24 18:44:17.751709 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.751690 2575 scope.go:117] "RemoveContainer" containerID="00866783f35039a445c9162658106a7ac6f3aa6ceda7fb64de7277bafb3e1849" Jul 24 18:44:17.752954 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.752931 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" event={"ID":"be74898d-3d60-437f-9c90-5408b5fbb3a3","Type":"ContainerStarted","Data":"895798cbf00458bd997008b9b179d08ec92761dde196233dd85ab728f46783b1"} Jul 24 18:44:17.765046 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.764990 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-6cl5p" podStartSLOduration=2.475878723 podStartE2EDuration="8.764972697s" totalImagesPullingTime="6.289093974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:17.763583513 +0000 UTC m=+52.840096089" watchObservedRunningTime="2026-07-24 18:44:17.764972697 +0000 UTC m=+52.841485256" Jul 24 18:44:17.782907 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.782846 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" podStartSLOduration=1.558807166 podStartE2EDuration="7.78282301s" totalImagesPullingTime="6.224015844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:17.781176521 +0000 UTC m=+52.857689082" watchObservedRunningTime="2026-07-24 18:44:17.78282301 +0000 UTC m=+52.859335580" Jul 24 18:44:17.801563 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.801492 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" podStartSLOduration=2.499471094 podStartE2EDuration="8.801470508s" totalImagesPullingTime="6.301999414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:17.800589266 +0000 UTC m=+52.877101852" watchObservedRunningTime="2026-07-24 18:44:17.801470508 +0000 UTC m=+52.877983066" Jul 24 18:44:17.842507 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.842384 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xvxgr" podStartSLOduration=2.494651165 podStartE2EDuration="8.842364378s" totalImagesPullingTime="6.347713213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:17.825489644 +0000 UTC m=+52.902002406" watchObservedRunningTime="2026-07-24 18:44:17.842364378 +0000 UTC m=+52.918876939" Jul 24 18:44:17.887129 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.887101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:17.887245 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:17.887230 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:17.887618 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:17.887597 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls podName:9ca07cf2-39b7-4962-ad44-fb73e63c4005 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:25.887574418 +0000 UTC m=+60.964086961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-qtf2r" (UID: "9ca07cf2-39b7-4962-ad44-fb73e63c4005") : secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:17.989111 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.989076 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:17.989251 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.989131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:17.989251 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:17.989201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:17.989342 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:17.989257 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 18:44:17.989342 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:17.989327 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 18:44:17.989660 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:17.989330 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:25.989309034 +0000 UTC m=+61.065821572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : configmap references non-existent config key: service-ca.crt Jul 24 18:44:17.989660 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:17.989382 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:25.98936624 +0000 UTC m=+61.065878782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : secret "router-metrics-certs-default" not found Jul 24 18:44:17.989660 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:17.989445 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls podName:31882160-593a-41c1-8b4a-9025178b589f nodeName:}" failed. No retries permitted until 2026-07-24 18:44:25.989435577 +0000 UTC m=+61.065948122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls") pod "cluster-samples-operator-6748489656-g8rc4" (UID: "31882160-593a-41c1-8b4a-9025178b589f") : secret "samples-operator-tls" not found Jul 24 18:44:18.758147 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.758120 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 18:44:18.758609 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.758595 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/0.log" Jul 24 18:44:18.758669 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.758627 2575 generic.go:356] "Generic (PLEG): container finished" podID="c683dae8-6f44-455b-bcd6-d81c0834ad39" containerID="652973601c05878c7a69c645647ac96ca1255e57c4d82a6a08869dbcb8695764" exitCode=255 Jul 24 18:44:18.758779 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.758759 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" event={"ID":"c683dae8-6f44-455b-bcd6-d81c0834ad39","Type":"ContainerDied","Data":"652973601c05878c7a69c645647ac96ca1255e57c4d82a6a08869dbcb8695764"} Jul 24 18:44:18.758845 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.758823 2575 scope.go:117] "RemoveContainer" containerID="00866783f35039a445c9162658106a7ac6f3aa6ceda7fb64de7277bafb3e1849" Jul 24 18:44:18.759059 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.759039 2575 scope.go:117] "RemoveContainer" containerID="652973601c05878c7a69c645647ac96ca1255e57c4d82a6a08869dbcb8695764" Jul 24 18:44:18.759241 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:18.759220 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-p2c9m_openshift-console-operator(c683dae8-6f44-455b-bcd6-d81c0834ad39)\"" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" podUID="c683dae8-6f44-455b-bcd6-d81c0834ad39" Jul 24 18:44:18.835765 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.835720 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c2ppz"] Jul 24 18:44:18.858586 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.858559 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2ppz"] Jul 24 18:44:18.858769 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.858692 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:18.861617 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.861587 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 18:44:18.861772 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.861709 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 18:44:18.861772 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.861587 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5dvls\"" Jul 24 18:44:18.898216 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.898186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:18.898348 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.898262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:18.898348 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.898304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pkc\" (UniqueName: \"kubernetes.io/projected/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-api-access-r5pkc\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:18.898348 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.898340 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85241c87-6a5d-4dce-bb06-9a46755ee37b-crio-socket\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:18.898465 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.898361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85241c87-6a5d-4dce-bb06-9a46755ee37b-data-volume\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:18.916278 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.916239 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:18.943492 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.943438 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv"] Jul 24 18:44:18.962509 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.962474 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv"] Jul 24 18:44:18.962647 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.962593 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv" Jul 24 18:44:18.965386 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.965351 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 24 18:44:18.965505 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.965399 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 24 18:44:18.965505 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.965432 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-g925r\"" Jul 24 18:44:18.999255 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.999223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:18.999255 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.999259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdw7\" (UniqueName: \"kubernetes.io/projected/4fe1e19b-8c9a-4915-8b6f-391712f0fed2-kube-api-access-vrdw7\") pod \"migrator-6865c9967-2fczv\" (UID: \"4fe1e19b-8c9a-4915-8b6f-391712f0fed2\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv" Jul 24 18:44:18.999472 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:18.999395 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 18:44:18.999472 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.999419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:18.999589 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:18.999485 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 18:44:18.999589 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:18.999496 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls podName:85241c87-6a5d-4dce-bb06-9a46755ee37b nodeName:}" failed. No retries permitted until 2026-07-24 18:44:19.499444646 +0000 UTC m=+54.575957187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2ppz" (UID: "85241c87-6a5d-4dce-bb06-9a46755ee37b") : secret "insights-runtime-extractor-tls" not found Jul 24 18:44:18.999829 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.999793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pkc\" (UniqueName: \"kubernetes.io/projected/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-api-access-r5pkc\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:18.999924 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:18.999891 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm podName:85241c87-6a5d-4dce-bb06-9a46755ee37b nodeName:}" failed. No retries permitted until 2026-07-24 18:44:19.499855621 +0000 UTC m=+54.576368169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2ppz" (UID: "85241c87-6a5d-4dce-bb06-9a46755ee37b") : configmap "kube-rbac-proxy" not found Jul 24 18:44:18.999987 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.999949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85241c87-6a5d-4dce-bb06-9a46755ee37b-crio-socket\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:19.000035 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:18.999988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85241c87-6a5d-4dce-bb06-9a46755ee37b-data-volume\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:19.000227 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.000198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85241c87-6a5d-4dce-bb06-9a46755ee37b-crio-socket\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:19.000365 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.000345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85241c87-6a5d-4dce-bb06-9a46755ee37b-data-volume\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:19.008781 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.008691 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:19.012170 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.012148 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pkc\" (UniqueName: \"kubernetes.io/projected/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-api-access-r5pkc\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:19.101205 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.101175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdw7\" (UniqueName: \"kubernetes.io/projected/4fe1e19b-8c9a-4915-8b6f-391712f0fed2-kube-api-access-vrdw7\") pod \"migrator-6865c9967-2fczv\" (UID: \"4fe1e19b-8c9a-4915-8b6f-391712f0fed2\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv" Jul 24 18:44:19.108727 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.108699 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:19.109398 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.109378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdw7\" (UniqueName: \"kubernetes.io/projected/4fe1e19b-8c9a-4915-8b6f-391712f0fed2-kube-api-access-vrdw7\") pod \"migrator-6865c9967-2fczv\" (UID: \"4fe1e19b-8c9a-4915-8b6f-391712f0fed2\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv" Jul 24 18:44:19.161791 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.161730 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:19.273043 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.272968 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv" Jul 24 18:44:19.372069 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.372033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:19.415588 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.415548 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv"] Jul 24 18:44:19.418484 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:19.418457 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe1e19b_8c9a_4915_8b6f_391712f0fed2.slice/crio-97eb5970febf0fe17fc89792547101f6727fb9a28e06f6c591bde3e5c04ee334 WatchSource:0}: Error finding container 97eb5970febf0fe17fc89792547101f6727fb9a28e06f6c591bde3e5c04ee334: Status 404 returned error can't find the container with id 97eb5970febf0fe17fc89792547101f6727fb9a28e06f6c591bde3e5c04ee334 Jul 24 18:44:19.504855 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.504820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:19.505022 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.504886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:19.505022 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:19.504986 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 18:44:19.505022 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:19.505012 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 18:44:19.505136 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:19.505063 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls podName:85241c87-6a5d-4dce-bb06-9a46755ee37b nodeName:}" failed. No retries permitted until 2026-07-24 18:44:20.5050463 +0000 UTC m=+55.581558839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2ppz" (UID: "85241c87-6a5d-4dce-bb06-9a46755ee37b") : secret "insights-runtime-extractor-tls" not found Jul 24 18:44:19.505136 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:19.505077 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm podName:85241c87-6a5d-4dce-bb06-9a46755ee37b nodeName:}" failed. No retries permitted until 2026-07-24 18:44:20.505070085 +0000 UTC m=+55.581582621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2ppz" (UID: "85241c87-6a5d-4dce-bb06-9a46755ee37b") : configmap "kube-rbac-proxy" not found Jul 24 18:44:19.762887 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.762858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv" event={"ID":"4fe1e19b-8c9a-4915-8b6f-391712f0fed2","Type":"ContainerStarted","Data":"97eb5970febf0fe17fc89792547101f6727fb9a28e06f6c591bde3e5c04ee334"} Jul 24 18:44:19.764350 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.764331 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 18:44:19.764727 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.764710 2575 scope.go:117] "RemoveContainer" containerID="652973601c05878c7a69c645647ac96ca1255e57c4d82a6a08869dbcb8695764" Jul 24 18:44:19.764957 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:19.764933 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-p2c9m_openshift-console-operator(c683dae8-6f44-455b-bcd6-d81c0834ad39)\"" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" podUID="c683dae8-6f44-455b-bcd6-d81c0834ad39" Jul 24 18:44:19.830154 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.830132 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pgx49_7d43cff0-e130-45e6-8fbc-e212008b30a4/dns-node-resolver/0.log" Jul 24 18:44:19.852607 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.852579 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:19.952759 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:19.952701 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:20.348721 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:20.348683 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:20.348981 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:20.348763 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:20.425083 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:20.425051 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s29nq_e6e2711c-af0a-41eb-b4d0-d1d86509fdc1/node-ca/0.log" Jul 24 18:44:20.515540 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:20.515498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:20.515711 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:20.515605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:20.515711 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:20.515673 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 18:44:20.515846 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:20.515759 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls podName:85241c87-6a5d-4dce-bb06-9a46755ee37b nodeName:}" failed. No retries permitted until 2026-07-24 18:44:22.515725305 +0000 UTC m=+57.592237861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2ppz" (UID: "85241c87-6a5d-4dce-bb06-9a46755ee37b") : secret "insights-runtime-extractor-tls" not found Jul 24 18:44:20.515846 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:20.515771 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 18:44:20.515846 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:20.515838 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm podName:85241c87-6a5d-4dce-bb06-9a46755ee37b nodeName:}" failed. No retries permitted until 2026-07-24 18:44:22.515819854 +0000 UTC m=+57.592332407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2ppz" (UID: "85241c87-6a5d-4dce-bb06-9a46755ee37b") : configmap "kube-rbac-proxy" not found Jul 24 18:44:20.768527 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:20.768497 2575 scope.go:117] "RemoveContainer" containerID="652973601c05878c7a69c645647ac96ca1255e57c4d82a6a08869dbcb8695764" Jul 24 18:44:20.768913 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:20.768663 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-p2c9m_openshift-console-operator(c683dae8-6f44-455b-bcd6-d81c0834ad39)\"" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" podUID="c683dae8-6f44-455b-bcd6-d81c0834ad39" Jul 24 18:44:21.025765 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.025670 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5hsw_249a8d0d-04b2-4720-a52d-37c37ff7a4ac/serve-healthcheck-canary/0.log" Jul 24 18:44:21.445515 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.445482 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-w4dkx"] Jul 24 18:44:21.462949 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.462923 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-w4dkx"] Jul 24 18:44:21.463114 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.463011 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.466297 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.466276 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 24 18:44:21.466439 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.466319 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 24 18:44:21.467246 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.467227 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 24 18:44:21.467362 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.467265 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 24 18:44:21.467362 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.467351 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6fj6q\"" Jul 24 18:44:21.525033 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.525003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5322a8-4243-44ac-bff3-182cfe6aa4c5-signing-cabundle\") pod \"service-ca-658d7b7c78-w4dkx\" (UID: \"8d5322a8-4243-44ac-bff3-182cfe6aa4c5\") " pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.525237 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.525205 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8d5322a8-4243-44ac-bff3-182cfe6aa4c5-signing-key\") pod \"service-ca-658d7b7c78-w4dkx\" (UID: \"8d5322a8-4243-44ac-bff3-182cfe6aa4c5\") " pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.525309 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.525258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-848km\" (UniqueName: \"kubernetes.io/projected/8d5322a8-4243-44ac-bff3-182cfe6aa4c5-kube-api-access-848km\") pod \"service-ca-658d7b7c78-w4dkx\" (UID: \"8d5322a8-4243-44ac-bff3-182cfe6aa4c5\") " pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.566182 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.566134 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:21.625624 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.625582 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8d5322a8-4243-44ac-bff3-182cfe6aa4c5-signing-key\") pod \"service-ca-658d7b7c78-w4dkx\" (UID: \"8d5322a8-4243-44ac-bff3-182cfe6aa4c5\") " pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.625849 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.625648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-848km\" (UniqueName: \"kubernetes.io/projected/8d5322a8-4243-44ac-bff3-182cfe6aa4c5-kube-api-access-848km\") pod \"service-ca-658d7b7c78-w4dkx\" (UID: \"8d5322a8-4243-44ac-bff3-182cfe6aa4c5\") " pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.625849 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.625707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5322a8-4243-44ac-bff3-182cfe6aa4c5-signing-cabundle\") pod \"service-ca-658d7b7c78-w4dkx\" (UID: \"8d5322a8-4243-44ac-bff3-182cfe6aa4c5\") " pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.626354 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.626335 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5322a8-4243-44ac-bff3-182cfe6aa4c5-signing-cabundle\") pod \"service-ca-658d7b7c78-w4dkx\" (UID: \"8d5322a8-4243-44ac-bff3-182cfe6aa4c5\") " pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.628240 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.628213 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/8d5322a8-4243-44ac-bff3-182cfe6aa4c5-signing-key\") pod \"service-ca-658d7b7c78-w4dkx\" (UID: \"8d5322a8-4243-44ac-bff3-182cfe6aa4c5\") " pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.634811 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.634790 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-848km\" (UniqueName: \"kubernetes.io/projected/8d5322a8-4243-44ac-bff3-182cfe6aa4c5-kube-api-access-848km\") pod \"service-ca-658d7b7c78-w4dkx\" (UID: \"8d5322a8-4243-44ac-bff3-182cfe6aa4c5\") " pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.666056 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.666020 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:21.771980 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.771895 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" Jul 24 18:44:21.773880 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.773732 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv" event={"ID":"4fe1e19b-8c9a-4915-8b6f-391712f0fed2","Type":"ContainerStarted","Data":"486c4e655debb3e2883e0671dcb4c8ee9a294fb8062d96d37a26e85919cf67f3"} Jul 24 18:44:21.773880 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.773804 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv" event={"ID":"4fe1e19b-8c9a-4915-8b6f-391712f0fed2","Type":"ContainerStarted","Data":"9c157048b90c2ba193055cc1f6a5c4b873e71b332042a90b0b6318ac99f0ec7b"} Jul 24 18:44:21.795078 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.793130 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-2fczv" podStartSLOduration=2.352587096 podStartE2EDuration="3.793093302s" totalImagesPullingTime="1.440506206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:21.790514706 +0000 UTC m=+56.867027260" watchObservedRunningTime="2026-07-24 18:44:21.793093302 +0000 UTC m=+56.869605863" Jul 24 18:44:21.869324 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.869277 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:21.902505 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.902476 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-w4dkx"] Jul 24 18:44:21.905599 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:21.905572 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5322a8_4243_44ac_bff3_182cfe6aa4c5.slice/crio-ea69d9107e7160912a2670beff19b0f1ab2b1f5758a7df2b8489ef31e6a5908a WatchSource:0}: Error finding container ea69d9107e7160912a2670beff19b0f1ab2b1f5758a7df2b8489ef31e6a5908a: Status 404 returned error can't find the container with id ea69d9107e7160912a2670beff19b0f1ab2b1f5758a7df2b8489ef31e6a5908a Jul 24 18:44:21.970102 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:21.970078 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:22.372174 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:22.372133 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:22.533755 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:22.533708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:22.533911 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:22.533810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:22.533911 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:22.533868 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 18:44:22.533982 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:22.533924 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 18:44:22.533982 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:22.533952 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls podName:85241c87-6a5d-4dce-bb06-9a46755ee37b nodeName:}" failed. No retries permitted until 2026-07-24 18:44:26.533931405 +0000 UTC m=+61.610443985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2ppz" (UID: "85241c87-6a5d-4dce-bb06-9a46755ee37b") : secret "insights-runtime-extractor-tls" not found Jul 24 18:44:22.533982 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:22.533971 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm podName:85241c87-6a5d-4dce-bb06-9a46755ee37b nodeName:}" failed. No retries permitted until 2026-07-24 18:44:26.533962064 +0000 UTC m=+61.610474602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2ppz" (UID: "85241c87-6a5d-4dce-bb06-9a46755ee37b") : configmap "kube-rbac-proxy" not found Jul 24 18:44:22.694250 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:22.694211 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:22.780724 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:22.780689 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" event={"ID":"8d5322a8-4243-44ac-bff3-182cfe6aa4c5","Type":"ContainerStarted","Data":"8af68cfafe01f12ebb3b4be592cda4d71c39148c53609546c77f374a474976b5"} Jul 24 18:44:22.781194 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:22.780819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" event={"ID":"8d5322a8-4243-44ac-bff3-182cfe6aa4c5","Type":"ContainerStarted","Data":"ea69d9107e7160912a2670beff19b0f1ab2b1f5758a7df2b8489ef31e6a5908a"} Jul 24 18:44:22.793444 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:22.793415 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:22.802117 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:22.802065 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-w4dkx" podStartSLOduration=1.802047318 podStartE2EDuration="1.802047318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:22.800708793 +0000 UTC m=+57.877221353" watchObservedRunningTime="2026-07-24 18:44:22.802047318 +0000 UTC m=+57.878559879" Jul 24 18:44:22.896681 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:22.896641 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:23.029694 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:23.029599 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:23.130164 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:23.130124 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:23.600320 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:23.600286 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:24.185195 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:24.185160 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:24.790906 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:24.790875 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:24.992725 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:24.992687 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:25.419245 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:25.419214 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:25.513674 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:25.513641 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:25.613609 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:25.613573 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:25.724925 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:25.724837 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:25.967842 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:25.967802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:25.968092 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:25.967955 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:25.968092 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:25.968028 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls podName:9ca07cf2-39b7-4962-ad44-fb73e63c4005 nodeName:}" failed. No retries permitted until 2026-07-24 18:44:41.968010857 +0000 UTC m=+77.044523392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-qtf2r" (UID: "9ca07cf2-39b7-4962-ad44-fb73e63c4005") : secret "cluster-monitoring-operator-tls" not found Jul 24 18:44:26.001639 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.001567 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:26.068455 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.068424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:26.068619 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.068463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:26.068619 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:26.068573 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:42.068558941 +0000 UTC m=+77.145071477 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : configmap references non-existent config key: service-ca.crt Jul 24 18:44:26.068619 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:26.068583 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 18:44:26.068769 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.068646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:26.068769 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:26.068659 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs podName:ecfae2bd-f425-407e-b825-d5505cd538ef nodeName:}" failed. No retries permitted until 2026-07-24 18:44:42.06864296 +0000 UTC m=+77.145155503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs") pod "router-default-6f4d68c8f7-vqjh4" (UID: "ecfae2bd-f425-407e-b825-d5505cd538ef") : secret "router-metrics-certs-default" not found Jul 24 18:44:26.071184 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.071153 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/31882160-593a-41c1-8b4a-9025178b589f-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-g8rc4\" (UID: \"31882160-593a-41c1-8b4a-9025178b589f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:26.330690 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.330615 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5c5v5\"" Jul 24 18:44:26.338929 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.338901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" Jul 24 18:44:26.392802 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.392170 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:26.473259 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.473228 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4"] Jul 24 18:44:26.574070 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.574044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:26.574197 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.574159 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:26.574197 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:26.574165 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 18:44:26.574271 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:26.574222 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm podName:85241c87-6a5d-4dce-bb06-9a46755ee37b nodeName:}" failed. No retries permitted until 2026-07-24 18:44:34.574206774 +0000 UTC m=+69.650719310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c2ppz" (UID: "85241c87-6a5d-4dce-bb06-9a46755ee37b") : configmap "kube-rbac-proxy" not found Jul 24 18:44:26.574271 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:26.574255 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 18:44:26.574342 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:44:26.574307 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls podName:85241c87-6a5d-4dce-bb06-9a46755ee37b nodeName:}" failed. No retries permitted until 2026-07-24 18:44:34.57429684 +0000 UTC m=+69.650809376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c2ppz" (UID: "85241c87-6a5d-4dce-bb06-9a46755ee37b") : secret "insights-runtime-extractor-tls" not found Jul 24 18:44:26.595961 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.595880 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:26.676770 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.676729 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8qzpr" Jul 24 18:44:26.695867 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.695842 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:26.795295 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.795265 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" event={"ID":"31882160-593a-41c1-8b4a-9025178b589f","Type":"ContainerStarted","Data":"f6e7b5e126d5e25e0cd6af7cac51a04127a095786644b38942f451878a0fab57"} Jul 24 18:44:26.993825 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:26.993795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:27.189147 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:27.189118 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:27.291078 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:27.290972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" limit=1024 Jul 24 18:44:28.807431 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:28.807397 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" event={"ID":"31882160-593a-41c1-8b4a-9025178b589f","Type":"ContainerStarted","Data":"71a0f39f31aef85b2649bea23b0ebffefcae2912534e3c3fc58cf0af7165b96d"} Jul 24 18:44:28.807880 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:28.807481 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" event={"ID":"31882160-593a-41c1-8b4a-9025178b589f","Type":"ContainerStarted","Data":"3e2eda5c5a2fd01a31cbd683920ecf7c133fae9074e0a86208e734991a0b6d02"} Jul 24 18:44:28.828261 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:28.828210 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-g8rc4" podStartSLOduration=17.00482351 podStartE2EDuration="18.828195146s" totalImagesPullingTime="1.823371636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:28.826977636 +0000 UTC m=+63.903490213" watchObservedRunningTime="2026-07-24 18:44:28.828195146 +0000 UTC m=+63.904707741" Jul 24 18:44:31.116015 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.115975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:44:31.118771 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.118735 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 18:44:31.129312 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.129278 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ed67684-a72a-4ec0-9a45-3871b53a3e72-metrics-certs\") pod \"network-metrics-daemon-khpvw\" (UID: \"3ed67684-a72a-4ec0-9a45-3871b53a3e72\") " pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:44:31.217163 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.217130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:44:31.219953 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.219925 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 18:44:31.229789 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.229760 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a976a8d-b47d-4dbb-a8cd-f9a7170875a8-original-pull-secret\") pod \"global-pull-secret-syncer-5q8wz\" (UID: \"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8\") " pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:44:31.318498 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.318456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qwq7\" (UniqueName: \"kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7\") pod \"network-check-target-x6l9k\" (UID: \"641810ec-5d18-41c3-990b-462e721aa175\") " pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:44:31.321048 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.321024 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qwq7\" (UniqueName: \"kubernetes.io/projected/641810ec-5d18-41c3-990b-462e721aa175-kube-api-access-5qwq7\") pod \"network-check-target-x6l9k\" (UID: \"641810ec-5d18-41c3-990b-462e721aa175\") " pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:44:31.425504 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.425479 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-96zrd\"" Jul 24 18:44:31.427623 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.427605 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5q8wz" Jul 24 18:44:31.433649 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.433632 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-khpvw" Jul 24 18:44:31.435162 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.435144 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6w7xn\"" Jul 24 18:44:31.443645 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.443620 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:44:31.586133 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.586098 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-khpvw"] Jul 24 18:44:31.590342 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:31.590312 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed67684_a72a_4ec0_9a45_3871b53a3e72.slice/crio-dcd551ebb7b5b7ab47a07f3d8a7d04094fa610f113564c6e6be7cb5278080277 WatchSource:0}: Error finding container dcd551ebb7b5b7ab47a07f3d8a7d04094fa610f113564c6e6be7cb5278080277: Status 404 returned error can't find the container with id dcd551ebb7b5b7ab47a07f3d8a7d04094fa610f113564c6e6be7cb5278080277 Jul 24 18:44:31.606039 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.606013 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5q8wz"] Jul 24 18:44:31.608351 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:31.608309 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a976a8d_b47d_4dbb_a8cd_f9a7170875a8.slice/crio-330751fe6399f344f4a1bd842b82620667c5fbb08dfe4e86d544b0f8777fe555 WatchSource:0}: Error finding container 330751fe6399f344f4a1bd842b82620667c5fbb08dfe4e86d544b0f8777fe555: Status 404 returned error can't find the container with id 330751fe6399f344f4a1bd842b82620667c5fbb08dfe4e86d544b0f8777fe555 Jul 24 18:44:31.621465 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.621439 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x6l9k"] Jul 24 18:44:31.623604 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:31.623574 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641810ec_5d18_41c3_990b_462e721aa175.slice/crio-9114f59919dd974dd4ef986908eb6aa2c33a20378aa1088364e34218ff987ce8 WatchSource:0}: Error finding container 9114f59919dd974dd4ef986908eb6aa2c33a20378aa1088364e34218ff987ce8: Status 404 returned error can't find the container with id 9114f59919dd974dd4ef986908eb6aa2c33a20378aa1088364e34218ff987ce8 Jul 24 18:44:31.817830 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.817791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-khpvw" event={"ID":"3ed67684-a72a-4ec0-9a45-3871b53a3e72","Type":"ContainerStarted","Data":"dcd551ebb7b5b7ab47a07f3d8a7d04094fa610f113564c6e6be7cb5278080277"} Jul 24 18:44:31.818987 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.818951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5q8wz" event={"ID":"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8","Type":"ContainerStarted","Data":"330751fe6399f344f4a1bd842b82620667c5fbb08dfe4e86d544b0f8777fe555"} Jul 24 18:44:31.820425 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.820406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x6l9k" event={"ID":"641810ec-5d18-41c3-990b-462e721aa175","Type":"ContainerStarted","Data":"785ffcd070508931670da77305fd0ce915606424cccfc0c600dc977967e63387"} Jul 24 18:44:31.820538 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.820441 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x6l9k" event={"ID":"641810ec-5d18-41c3-990b-462e721aa175","Type":"ContainerStarted","Data":"9114f59919dd974dd4ef986908eb6aa2c33a20378aa1088364e34218ff987ce8"} Jul 24 18:44:31.820538 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.820477 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:44:31.836085 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:31.836037 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x6l9k" podStartSLOduration=66.836020225 podStartE2EDuration="1m6.836020225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:31.835573368 +0000 UTC m=+66.912085931" watchObservedRunningTime="2026-07-24 18:44:31.836020225 +0000 UTC m=+66.912532784" Jul 24 18:44:32.226326 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.226293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:32.226700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.226407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:32.229158 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.228983 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0627a68a-b4bf-4eed-ae3d-6a3d4f97505e-metrics-tls\") pod \"dns-default-54h5x\" (UID: \"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e\") " pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:32.229304 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.229198 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") pod \"image-registry-5d954f5747-7kxdd\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:32.409936 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.409898 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4z6xk\"" Jul 24 18:44:32.410515 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.410495 2575 scope.go:117] "RemoveContainer" containerID="652973601c05878c7a69c645647ac96ca1255e57c4d82a6a08869dbcb8695764" Jul 24 18:44:32.414062 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.414044 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:32.427884 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.427851 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fjp2z\"" Jul 24 18:44:32.435165 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.435143 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:32.629700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.629617 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d954f5747-7kxdd"] Jul 24 18:44:32.636354 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:32.636325 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425fa486_2440_4be2_b6fb_34f609ab97d5.slice/crio-c30e7c388c4c859c2d75f084aa3697b8a4b02144d635daa28d11eebd95531119 WatchSource:0}: Error finding container c30e7c388c4c859c2d75f084aa3697b8a4b02144d635daa28d11eebd95531119: Status 404 returned error can't find the container with id c30e7c388c4c859c2d75f084aa3697b8a4b02144d635daa28d11eebd95531119 Jul 24 18:44:32.656073 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.656007 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-54h5x"] Jul 24 18:44:32.659596 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:32.659558 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0627a68a_b4bf_4eed_ae3d_6a3d4f97505e.slice/crio-153dbe897dc8bca8606bf1a154edf051891d3fcfac18919cf116f95c64271c55 WatchSource:0}: Error finding container 153dbe897dc8bca8606bf1a154edf051891d3fcfac18919cf116f95c64271c55: Status 404 returned error can't find the container with id 153dbe897dc8bca8606bf1a154edf051891d3fcfac18919cf116f95c64271c55 Jul 24 18:44:32.826470 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.826426 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" event={"ID":"425fa486-2440-4be2-b6fb-34f609ab97d5","Type":"ContainerStarted","Data":"2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691"} Jul 24 18:44:32.826643 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.826485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" event={"ID":"425fa486-2440-4be2-b6fb-34f609ab97d5","Type":"ContainerStarted","Data":"c30e7c388c4c859c2d75f084aa3697b8a4b02144d635daa28d11eebd95531119"} Jul 24 18:44:32.826702 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.826644 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:32.828846 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.828821 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 18:44:32.828974 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.828923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" event={"ID":"c683dae8-6f44-455b-bcd6-d81c0834ad39","Type":"ContainerStarted","Data":"ab24ab584adbd26690931c796470564d0bb8030c8c9c2f7b9197cff382cbe650"} Jul 24 18:44:32.829209 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.829151 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:32.831595 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.831560 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54h5x" event={"ID":"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e","Type":"ContainerStarted","Data":"153dbe897dc8bca8606bf1a154edf051891d3fcfac18919cf116f95c64271c55"} Jul 24 18:44:32.848079 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.846349 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" podStartSLOduration=66.846325043 podStartE2EDuration="1m6.846325043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:32.846083875 +0000 UTC m=+67.922596462" watchObservedRunningTime="2026-07-24 18:44:32.846325043 +0000 UTC m=+67.922837603" Jul 24 18:44:32.863835 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.863763 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" podStartSLOduration=17.595517878 podStartE2EDuration="23.863716065s" totalImagesPullingTime="6.268198187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:32.862823698 +0000 UTC m=+67.939336281" watchObservedRunningTime="2026-07-24 18:44:32.863716065 +0000 UTC m=+67.940228626" Jul 24 18:44:32.928498 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:32.928468 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-p2c9m" Jul 24 18:44:33.840702 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:33.840638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-khpvw" event={"ID":"3ed67684-a72a-4ec0-9a45-3871b53a3e72","Type":"ContainerStarted","Data":"2afbfc95065af1dd108c573985f03b8fd155a74610b9d8bd5c173b3a7d0464fe"} Jul 24 18:44:34.648665 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:34.648624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:34.648858 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:34.648714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:34.649283 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:34.649259 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85241c87-6a5d-4dce-bb06-9a46755ee37b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:34.651450 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:34.651424 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85241c87-6a5d-4dce-bb06-9a46755ee37b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c2ppz\" (UID: \"85241c87-6a5d-4dce-bb06-9a46755ee37b\") " pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:34.771686 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:34.771651 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5dvls\"" Jul 24 18:44:34.779105 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:34.779070 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c2ppz" Jul 24 18:44:34.848473 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:34.848426 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-khpvw" event={"ID":"3ed67684-a72a-4ec0-9a45-3871b53a3e72","Type":"ContainerStarted","Data":"10aab4a027400c9f0a4acadbd95e4d54badf66c7bb0e93115b59ed52142eb6dc"} Jul 24 18:44:34.866595 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:34.866543 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-khpvw" podStartSLOduration=67.907933944 podStartE2EDuration="1m9.866527195s" totalImagesPullingTime="1.958593251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:34.864985829 +0000 UTC m=+69.941498390" watchObservedRunningTime="2026-07-24 18:44:34.866527195 +0000 UTC m=+69.943039771" Jul 24 18:44:36.595054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:36.595021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c2ppz"] Jul 24 18:44:36.599020 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:36.598996 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85241c87_6a5d_4dce_bb06_9a46755ee37b.slice/crio-826d23abbd244292b893328e5c7ce6018fbc607b47a83ff82b8f28c301d018c4 WatchSource:0}: Error finding container 826d23abbd244292b893328e5c7ce6018fbc607b47a83ff82b8f28c301d018c4: Status 404 returned error can't find the container with id 826d23abbd244292b893328e5c7ce6018fbc607b47a83ff82b8f28c301d018c4 Jul 24 18:44:36.856359 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:36.856266 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2ppz" event={"ID":"85241c87-6a5d-4dce-bb06-9a46755ee37b","Type":"ContainerStarted","Data":"826d23abbd244292b893328e5c7ce6018fbc607b47a83ff82b8f28c301d018c4"} Jul 24 18:44:36.857658 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:36.857630 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5q8wz" event={"ID":"2a976a8d-b47d-4dbb-a8cd-f9a7170875a8","Type":"ContainerStarted","Data":"efac5adc5f8e3511eeaea813aeb72cf5b821a84e3f983adb11d6749a762d990f"} Jul 24 18:44:36.859386 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:36.859364 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54h5x" event={"ID":"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e","Type":"ContainerStarted","Data":"fe2826479b3ca0c343116c33b3b2f5ac1aa4422a9d498f767db52ec641a0da90"} Jul 24 18:44:36.859468 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:36.859409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54h5x" event={"ID":"0627a68a-b4bf-4eed-ae3d-6a3d4f97505e","Type":"ContainerStarted","Data":"a492c454e6ce14e66decf96d08a5eea71d0e90cfd30974706a80b1edc8c2897a"} Jul 24 18:44:36.859519 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:36.859502 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:36.873481 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:36.873434 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5q8wz" podStartSLOduration=67.030969709 podStartE2EDuration="1m11.873419633s" totalImagesPullingTime="4.842449924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:36.872348627 +0000 UTC m=+71.948861196" watchObservedRunningTime="2026-07-24 18:44:36.873419633 +0000 UTC m=+71.949932191" Jul 24 18:44:36.891079 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:36.891029 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-54h5x" podStartSLOduration=33.101383141 podStartE2EDuration="36.891012027s" totalImagesPullingTime="3.789628886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:36.890828593 +0000 UTC m=+71.967341165" watchObservedRunningTime="2026-07-24 18:44:36.891012027 +0000 UTC m=+71.967524586" Jul 24 18:44:41.885434 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:41.885357 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2ppz" event={"ID":"85241c87-6a5d-4dce-bb06-9a46755ee37b","Type":"ContainerStarted","Data":"2679eaff0798810f7214465dc019319b90cdb71869823c39b22c5efe25d95bed"} Jul 24 18:44:42.015389 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.015354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:42.018289 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.018265 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ca07cf2-39b7-4962-ad44-fb73e63c4005-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-qtf2r\" (UID: \"9ca07cf2-39b7-4962-ad44-fb73e63c4005\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:42.116266 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.116222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:42.116429 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.116278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:42.117102 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.117075 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ecfae2bd-f425-407e-b825-d5505cd538ef-service-ca-bundle\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:42.119300 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.119275 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfae2bd-f425-407e-b825-d5505cd538ef-metrics-certs\") pod \"router-default-6f4d68c8f7-vqjh4\" (UID: \"ecfae2bd-f425-407e-b825-d5505cd538ef\") " pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:42.189557 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.189365 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x7xg2\"" Jul 24 18:44:42.196835 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.196799 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" Jul 24 18:44:42.252555 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.252523 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zh4sc\"" Jul 24 18:44:42.260925 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.260898 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:42.351542 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.351510 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r"] Jul 24 18:44:42.354714 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:42.354691 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca07cf2_39b7_4962_ad44_fb73e63c4005.slice/crio-ddebc2911810b98428b2b0d70505826ad220665fbacfdd419b319c5cfce71e77 WatchSource:0}: Error finding container ddebc2911810b98428b2b0d70505826ad220665fbacfdd419b319c5cfce71e77: Status 404 returned error can't find the container with id ddebc2911810b98428b2b0d70505826ad220665fbacfdd419b319c5cfce71e77 Jul 24 18:44:42.415824 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.415790 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f4d68c8f7-vqjh4"] Jul 24 18:44:42.420594 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:42.420566 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfae2bd_f425_407e_b825_d5505cd538ef.slice/crio-d2c812aa8fafc08e0b4ad719233a6f719b594a1ec5843db0555f6f1cb53ad927 WatchSource:0}: Error finding container d2c812aa8fafc08e0b4ad719233a6f719b594a1ec5843db0555f6f1cb53ad927: Status 404 returned error can't find the container with id d2c812aa8fafc08e0b4ad719233a6f719b594a1ec5843db0555f6f1cb53ad927 Jul 24 18:44:42.891827 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.891782 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" event={"ID":"ecfae2bd-f425-407e-b825-d5505cd538ef","Type":"ContainerStarted","Data":"f2202efc21d6f6c1085bdc4ff2dec0a4fc8f90a73a94f90d3f4c878ce87444e1"} Jul 24 18:44:42.892241 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.891849 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" event={"ID":"ecfae2bd-f425-407e-b825-d5505cd538ef","Type":"ContainerStarted","Data":"d2c812aa8fafc08e0b4ad719233a6f719b594a1ec5843db0555f6f1cb53ad927"} Jul 24 18:44:42.893800 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.893770 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" event={"ID":"9ca07cf2-39b7-4962-ad44-fb73e63c4005","Type":"ContainerStarted","Data":"ddebc2911810b98428b2b0d70505826ad220665fbacfdd419b319c5cfce71e77"} Jul 24 18:44:42.915970 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:42.915904 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" podStartSLOduration=32.915888353 podStartE2EDuration="32.915888353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:42.913353721 +0000 UTC m=+77.989866284" watchObservedRunningTime="2026-07-24 18:44:42.915888353 +0000 UTC m=+77.992400908" Jul 24 18:44:43.261412 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:43.261326 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:43.264516 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:43.264490 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:43.900759 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:43.900701 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:43.902062 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:43.902040 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6f4d68c8f7-vqjh4" Jul 24 18:44:45.842195 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.842119 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn"] Jul 24 18:44:45.847516 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.847492 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf"] Jul 24 18:44:45.847675 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.847630 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" Jul 24 18:44:45.851882 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.851799 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 24 18:44:45.856359 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.853262 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:45.856359 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.853789 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 18:44:45.856359 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.853868 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 18:44:45.856359 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.853789 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jptcv\"" Jul 24 18:44:45.856359 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.854030 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 18:44:45.856359 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.855575 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld"] Jul 24 18:44:45.857302 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.856919 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 18:44:45.857302 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.856920 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 18:44:45.857525 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.857504 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 18:44:45.857646 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.857546 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 18:44:45.859289 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.859265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:45.862055 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.862038 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 18:44:45.873772 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.873327 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn"] Jul 24 18:44:45.875301 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.875263 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf"] Jul 24 18:44:45.878267 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.878201 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld"] Jul 24 18:44:45.911676 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.911606 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2ppz" event={"ID":"85241c87-6a5d-4dce-bb06-9a46755ee37b","Type":"ContainerStarted","Data":"5cb04e56b77ea2e5f2598bfbe127e56073e95a1b2ccc7f2b9a60d326264cc3d5"} Jul 24 18:44:45.914668 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.914246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" event={"ID":"9ca07cf2-39b7-4962-ad44-fb73e63c4005","Type":"ContainerStarted","Data":"cda101a715d7a962fd8323242712e99b253f354647562570e512bbe37d816151"} Jul 24 18:44:45.945454 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.945369 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk"] Jul 24 18:44:45.952021 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.951997 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-xbdq4"] Jul 24 18:44:45.952153 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.952116 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" Jul 24 18:44:45.955113 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.954911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-hub\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:45.955594 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.955565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14469c5-770f-44eb-85fe-855ee6b69488-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:45.955769 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.955615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-ca\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:45.955769 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.955655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:45.955769 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.955678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:45.955769 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.955701 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13-tmp\") pod \"klusterlet-addon-workmgr-7c7b8558b8-4d4ld\" (UID: \"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:45.955769 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.955730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgsnz\" (UniqueName: \"kubernetes.io/projected/3466077f-4bdf-4240-ae5d-39226b08ad47-kube-api-access-tgsnz\") pod \"managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn\" (UID: \"3466077f-4bdf-4240-ae5d-39226b08ad47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" Jul 24 18:44:45.956027 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.955779 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3466077f-4bdf-4240-ae5d-39226b08ad47-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn\" (UID: \"3466077f-4bdf-4240-ae5d-39226b08ad47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" Jul 24 18:44:45.956027 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.955836 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh44r\" (UniqueName: \"kubernetes.io/projected/b14469c5-770f-44eb-85fe-855ee6b69488-kube-api-access-mh44r\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:45.956027 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.955861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c7b8558b8-4d4ld\" (UID: \"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:45.956027 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.955883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crf4\" (UniqueName: \"kubernetes.io/projected/b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13-kube-api-access-2crf4\") pod \"klusterlet-addon-workmgr-7c7b8558b8-4d4ld\" (UID: \"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:45.956206 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.956177 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xbdq4" Jul 24 18:44:45.957290 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.957269 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 18:44:45.957530 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.957376 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mj48f\"" Jul 24 18:44:45.957644 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.957625 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 18:44:45.958645 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.958626 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 18:44:45.958839 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.958717 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 18:44:45.962796 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.962732 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7kncp\"" Jul 24 18:44:45.970376 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.970352 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xbdq4"] Jul 24 18:44:45.979145 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:45.979120 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk"] Jul 24 18:44:46.034630 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.034580 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-qtf2r" podStartSLOduration=32.814218045 podStartE2EDuration="36.03456423s" totalImagesPullingTime="3.220346185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:46.033418395 +0000 UTC m=+81.109930954" watchObservedRunningTime="2026-07-24 18:44:46.03456423 +0000 UTC m=+81.111076784" Jul 24 18:44:46.056495 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.056458 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-ca\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.056648 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.056524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.056648 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.056551 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.056648 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.056574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13-tmp\") pod \"klusterlet-addon-workmgr-7c7b8558b8-4d4ld\" (UID: \"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:46.056648 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.056633 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgsnz\" (UniqueName: \"kubernetes.io/projected/3466077f-4bdf-4240-ae5d-39226b08ad47-kube-api-access-tgsnz\") pod \"managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn\" (UID: \"3466077f-4bdf-4240-ae5d-39226b08ad47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" Jul 24 18:44:46.056984 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.056671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3466077f-4bdf-4240-ae5d-39226b08ad47-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn\" (UID: \"3466077f-4bdf-4240-ae5d-39226b08ad47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" Jul 24 18:44:46.056984 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.056791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh44r\" (UniqueName: \"kubernetes.io/projected/b14469c5-770f-44eb-85fe-855ee6b69488-kube-api-access-mh44r\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.056984 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.056819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c7b8558b8-4d4ld\" (UID: \"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:46.056984 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.056844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crf4\" (UniqueName: \"kubernetes.io/projected/b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13-kube-api-access-2crf4\") pod \"klusterlet-addon-workmgr-7c7b8558b8-4d4ld\" (UID: \"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:46.056984 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.056916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2fbf38d4-922e-494d-ae19-415d297e9f14-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6fjlk\" (UID: \"2fbf38d4-922e-494d-ae19-415d297e9f14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" Jul 24 18:44:46.057236 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.057001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nqg\" (UniqueName: \"kubernetes.io/projected/88541e12-5e3b-47cd-947d-599b1501c178-kube-api-access-q7nqg\") pod \"downloads-7c97dc4555-xbdq4\" (UID: \"88541e12-5e3b-47cd-947d-599b1501c178\") " pod="openshift-console/downloads-7c97dc4555-xbdq4" Jul 24 18:44:46.057236 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.057064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-hub\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.057236 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.057091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fbf38d4-922e-494d-ae19-415d297e9f14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6fjlk\" (UID: \"2fbf38d4-922e-494d-ae19-415d297e9f14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" Jul 24 18:44:46.057236 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.057137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14469c5-770f-44eb-85fe-855ee6b69488-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.057996 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.057971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13-tmp\") pod \"klusterlet-addon-workmgr-7c7b8558b8-4d4ld\" (UID: \"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:46.058141 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.058121 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b14469c5-770f-44eb-85fe-855ee6b69488-additional-service-ca\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.060432 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.060384 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-hub\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.060529 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.060437 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13-klusterlet-config\") pod \"klusterlet-addon-workmgr-7c7b8558b8-4d4ld\" (UID: \"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:46.060599 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.060552 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.060875 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.060856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.061095 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.061076 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3466077f-4bdf-4240-ae5d-39226b08ad47-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn\" (UID: \"3466077f-4bdf-4240-ae5d-39226b08ad47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" Jul 24 18:44:46.061260 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.061241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b14469c5-770f-44eb-85fe-855ee6b69488-ca\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.073617 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.073594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgsnz\" (UniqueName: \"kubernetes.io/projected/3466077f-4bdf-4240-ae5d-39226b08ad47-kube-api-access-tgsnz\") pod \"managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn\" (UID: \"3466077f-4bdf-4240-ae5d-39226b08ad47\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" Jul 24 18:44:46.074497 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.074475 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crf4\" (UniqueName: \"kubernetes.io/projected/b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13-kube-api-access-2crf4\") pod \"klusterlet-addon-workmgr-7c7b8558b8-4d4ld\" (UID: \"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:46.074772 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.074727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh44r\" (UniqueName: \"kubernetes.io/projected/b14469c5-770f-44eb-85fe-855ee6b69488-kube-api-access-mh44r\") pod \"cluster-proxy-proxy-agent-b7b9785b-csbzf\" (UID: \"b14469c5-770f-44eb-85fe-855ee6b69488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.158472 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.158369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2fbf38d4-922e-494d-ae19-415d297e9f14-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6fjlk\" (UID: \"2fbf38d4-922e-494d-ae19-415d297e9f14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" Jul 24 18:44:46.158472 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.158449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nqg\" (UniqueName: \"kubernetes.io/projected/88541e12-5e3b-47cd-947d-599b1501c178-kube-api-access-q7nqg\") pod \"downloads-7c97dc4555-xbdq4\" (UID: \"88541e12-5e3b-47cd-947d-599b1501c178\") " pod="openshift-console/downloads-7c97dc4555-xbdq4" Jul 24 18:44:46.158779 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.158480 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fbf38d4-922e-494d-ae19-415d297e9f14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6fjlk\" (UID: \"2fbf38d4-922e-494d-ae19-415d297e9f14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" Jul 24 18:44:46.159540 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.159512 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2fbf38d4-922e-494d-ae19-415d297e9f14-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-6fjlk\" (UID: \"2fbf38d4-922e-494d-ae19-415d297e9f14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" Jul 24 18:44:46.161626 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.161599 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2fbf38d4-922e-494d-ae19-415d297e9f14-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-6fjlk\" (UID: \"2fbf38d4-922e-494d-ae19-415d297e9f14\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" Jul 24 18:44:46.174582 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.174560 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nqg\" (UniqueName: \"kubernetes.io/projected/88541e12-5e3b-47cd-947d-599b1501c178-kube-api-access-q7nqg\") pod \"downloads-7c97dc4555-xbdq4\" (UID: \"88541e12-5e3b-47cd-947d-599b1501c178\") " pod="openshift-console/downloads-7c97dc4555-xbdq4" Jul 24 18:44:46.176636 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.176613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" Jul 24 18:44:46.184797 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.184777 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" Jul 24 18:44:46.203934 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.203910 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:46.241710 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.241676 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v"] Jul 24 18:44:46.258007 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.257979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v" Jul 24 18:44:46.262164 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.262120 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 24 18:44:46.262394 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.262376 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hx9s5\"" Jul 24 18:44:46.265156 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.264962 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" Jul 24 18:44:46.270699 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.269291 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v"] Jul 24 18:44:46.272463 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.272171 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-xbdq4" Jul 24 18:44:46.361955 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.361235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1769c8bd-be29-4c80-b446-523184de4032-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-fvt2v\" (UID: \"1769c8bd-be29-4c80-b446-523184de4032\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v" Jul 24 18:44:46.416686 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.412180 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn"] Jul 24 18:44:46.450535 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.450425 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf"] Jul 24 18:44:46.454880 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:46.454845 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb14469c5_770f_44eb_85fe_855ee6b69488.slice/crio-1b8b0b76223decfa74c0ee183fcf7dc452d0ccce3b2de6f3d0e8acca08f6fe82 WatchSource:0}: Error finding container 1b8b0b76223decfa74c0ee183fcf7dc452d0ccce3b2de6f3d0e8acca08f6fe82: Status 404 returned error can't find the container with id 1b8b0b76223decfa74c0ee183fcf7dc452d0ccce3b2de6f3d0e8acca08f6fe82 Jul 24 18:44:46.462421 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.462386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1769c8bd-be29-4c80-b446-523184de4032-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-fvt2v\" (UID: \"1769c8bd-be29-4c80-b446-523184de4032\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v" Jul 24 18:44:46.465629 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.465602 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1769c8bd-be29-4c80-b446-523184de4032-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-fvt2v\" (UID: \"1769c8bd-be29-4c80-b446-523184de4032\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v" Jul 24 18:44:46.477415 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.477391 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld"] Jul 24 18:44:46.479511 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:46.479483 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb154ba0d_dbc5_4afc_a0a7_3e8e9bc92e13.slice/crio-c0c935e49be0569f1b5a0897d730b06899c69cb30386568702bec7ddee708382 WatchSource:0}: Error finding container c0c935e49be0569f1b5a0897d730b06899c69cb30386568702bec7ddee708382: Status 404 returned error can't find the container with id c0c935e49be0569f1b5a0897d730b06899c69cb30386568702bec7ddee708382 Jul 24 18:44:46.503345 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.503263 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk"] Jul 24 18:44:46.506066 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:46.506040 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbf38d4_922e_494d_ae19_415d297e9f14.slice/crio-216fe94fd65890d9328ab176ecf9ac290fbecd14ed9f96060f8da5d528ae1af6 WatchSource:0}: Error finding container 216fe94fd65890d9328ab176ecf9ac290fbecd14ed9f96060f8da5d528ae1af6: Status 404 returned error can't find the container with id 216fe94fd65890d9328ab176ecf9ac290fbecd14ed9f96060f8da5d528ae1af6 Jul 24 18:44:46.527005 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.526979 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-xbdq4"] Jul 24 18:44:46.531028 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:46.531004 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88541e12_5e3b_47cd_947d_599b1501c178.slice/crio-d4225ff86e21b57b4efdb195da08c74cd4c533204b5ae14afb34ce61c195a7c5 WatchSource:0}: Error finding container d4225ff86e21b57b4efdb195da08c74cd4c533204b5ae14afb34ce61c195a7c5: Status 404 returned error can't find the container with id d4225ff86e21b57b4efdb195da08c74cd4c533204b5ae14afb34ce61c195a7c5 Jul 24 18:44:46.577284 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.577258 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v" Jul 24 18:44:46.744940 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.744906 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v"] Jul 24 18:44:46.865563 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.865533 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-54h5x" Jul 24 18:44:46.921073 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.921037 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v" event={"ID":"1769c8bd-be29-4c80-b446-523184de4032","Type":"ContainerStarted","Data":"5a4ea78761c93856521cbc9a2258bd87ec77c409714cb1195326a8908d93de2b"} Jul 24 18:44:46.922723 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.922680 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" event={"ID":"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13","Type":"ContainerStarted","Data":"c0c935e49be0569f1b5a0897d730b06899c69cb30386568702bec7ddee708382"} Jul 24 18:44:46.924297 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.924269 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" event={"ID":"b14469c5-770f-44eb-85fe-855ee6b69488","Type":"ContainerStarted","Data":"1b8b0b76223decfa74c0ee183fcf7dc452d0ccce3b2de6f3d0e8acca08f6fe82"} Jul 24 18:44:46.925982 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.925950 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" event={"ID":"3466077f-4bdf-4240-ae5d-39226b08ad47","Type":"ContainerStarted","Data":"46f08e7cf51fb1af8ccbcf0f02e66a7f634079efeef976911e58031cbf6eeb83"} Jul 24 18:44:46.927470 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.927444 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xbdq4" event={"ID":"88541e12-5e3b-47cd-947d-599b1501c178","Type":"ContainerStarted","Data":"d4225ff86e21b57b4efdb195da08c74cd4c533204b5ae14afb34ce61c195a7c5"} Jul 24 18:44:46.930793 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:46.930193 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" event={"ID":"2fbf38d4-922e-494d-ae19-415d297e9f14","Type":"ContainerStarted","Data":"216fe94fd65890d9328ab176ecf9ac290fbecd14ed9f96060f8da5d528ae1af6"} Jul 24 18:44:50.046502 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.044997 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8d9fd6769-k4ggz"] Jul 24 18:44:50.057820 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.056733 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.061775 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.060894 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d9fd6769-k4ggz"] Jul 24 18:44:50.065520 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.064566 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rw2c6\"" Jul 24 18:44:50.065520 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.064779 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 18:44:50.065520 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.064931 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 18:44:50.069896 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.067016 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 18:44:50.069896 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.067032 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 18:44:50.069896 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.067198 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 18:44:50.199182 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.198496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-oauth-config\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.199517 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.199333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-service-ca\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.199517 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.199395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfcb\" (UniqueName: \"kubernetes.io/projected/65f99a57-f908-4e4c-a047-0e3d0b5fce74-kube-api-access-rbfcb\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.199517 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.199481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-serving-cert\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.199711 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.199568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-config\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.199711 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.199693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-oauth-serving-cert\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.300707 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.300597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-serving-cert\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.300707 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.300691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-config\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.300943 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.300770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-oauth-serving-cert\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.300943 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.300835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-oauth-config\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.300943 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.300863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-service-ca\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.300943 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.300923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfcb\" (UniqueName: \"kubernetes.io/projected/65f99a57-f908-4e4c-a047-0e3d0b5fce74-kube-api-access-rbfcb\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.307004 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.306816 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-config\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.307004 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.306938 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-oauth-serving-cert\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.307423 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.307400 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-service-ca\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.310593 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.310571 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfcb\" (UniqueName: \"kubernetes.io/projected/65f99a57-f908-4e4c-a047-0e3d0b5fce74-kube-api-access-rbfcb\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.316007 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.315974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-oauth-config\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.317345 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.317300 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-serving-cert\") pod \"console-8d9fd6769-k4ggz\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:50.373605 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:50.373559 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:44:52.421325 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:52.421280 2575 patch_prober.go:28] interesting pod/image-registry-5d954f5747-7kxdd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 18:44:52.421797 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:52.421392 2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" podUID="425fa486-2440-4be2-b6fb-34f609ab97d5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 18:44:53.845578 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:53.845550 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:44:57.610771 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:57.610680 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d9fd6769-k4ggz"] Jul 24 18:44:57.622303 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:44:57.615621 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f99a57_f908_4e4c_a047_0e3d0b5fce74.slice/crio-82ea93e55d198c216fd77f2d859f6815503b3bcaf23d4f018451ff74eafb3021 WatchSource:0}: Error finding container 82ea93e55d198c216fd77f2d859f6815503b3bcaf23d4f018451ff74eafb3021: Status 404 returned error can't find the container with id 82ea93e55d198c216fd77f2d859f6815503b3bcaf23d4f018451ff74eafb3021 Jul 24 18:44:58.000397 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.000309 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" event={"ID":"2fbf38d4-922e-494d-ae19-415d297e9f14","Type":"ContainerStarted","Data":"ee8d3448503f424e63e152d04f6c19a7d584557ccfa859c93030f38103f58f5f"} Jul 24 18:44:58.002280 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.002252 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v" event={"ID":"1769c8bd-be29-4c80-b446-523184de4032","Type":"ContainerStarted","Data":"be60b7ccb9c43987a13baa0408e18dce4305ef63a1b1c9abc292b3d76efb324e"} Jul 24 18:44:58.002452 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.002308 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v" Jul 24 18:44:58.003980 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.003957 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" event={"ID":"b154ba0d-dbc5-4afc-a0a7-3e8e9bc92e13","Type":"ContainerStarted","Data":"b1aad1f78db8953a9122ae4fe31e93d26c6a99ce6b6393cddc68a769768650f9"} Jul 24 18:44:58.004086 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.004008 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:58.006204 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.006184 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" Jul 24 18:44:58.006939 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.006915 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c2ppz" event={"ID":"85241c87-6a5d-4dce-bb06-9a46755ee37b","Type":"ContainerStarted","Data":"29846c2786f9cbaca35461c66a2c37ef304656aa58abb7bd1b7f38c57d30d8d3"} Jul 24 18:44:58.008118 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.008023 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v" Jul 24 18:44:58.009877 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.009855 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" event={"ID":"b14469c5-770f-44eb-85fe-855ee6b69488","Type":"ContainerStarted","Data":"4c3639d1d40e8b6fbb59e1947044e3077fa12208b5d73497402697a6449ec6ef"} Jul 24 18:44:58.009877 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.009888 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" event={"ID":"b14469c5-770f-44eb-85fe-855ee6b69488","Type":"ContainerStarted","Data":"50416aa6d6bc7eb852204d6c7f1bd7774b3acbf840f8a514cc74bf0f66f20a6a"} Jul 24 18:44:58.009877 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.009907 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" event={"ID":"b14469c5-770f-44eb-85fe-855ee6b69488","Type":"ContainerStarted","Data":"72969a9a418246022585f25ab6e266392bdbdcd1ab592850340d796a4178c315"} Jul 24 18:44:58.011079 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.011057 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d9fd6769-k4ggz" event={"ID":"65f99a57-f908-4e4c-a047-0e3d0b5fce74","Type":"ContainerStarted","Data":"82ea93e55d198c216fd77f2d859f6815503b3bcaf23d4f018451ff74eafb3021"} Jul 24 18:44:58.012525 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.012506 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" event={"ID":"3466077f-4bdf-4240-ae5d-39226b08ad47","Type":"ContainerStarted","Data":"e67374baf2791e587b16fa30a3871ae1ce5057f13c26e2f85fdd2193cab9a895"} Jul 24 18:44:58.017448 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.017392 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-6fjlk" podStartSLOduration=2.110093356 podStartE2EDuration="13.01737497s" totalImagesPullingTime="10.907281614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:58.016952707 +0000 UTC m=+93.093465266" watchObservedRunningTime="2026-07-24 18:44:58.01737497 +0000 UTC m=+93.093887527" Jul 24 18:44:58.039678 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.039612 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c2ppz" podStartSLOduration=19.343290718 podStartE2EDuration="40.039592684s" totalImagesPullingTime="20.696301966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:44:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:58.038610429 +0000 UTC m=+93.115122986" watchObservedRunningTime="2026-07-24 18:44:58.039592684 +0000 UTC m=+93.116105243" Jul 24 18:44:58.062501 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.062446 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dcd8d66c4-mpdqn" podStartSLOduration=2.075458151 podStartE2EDuration="13.062424466s" totalImagesPullingTime="10.986966315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:58.060385164 +0000 UTC m=+93.136897723" watchObservedRunningTime="2026-07-24 18:44:58.062424466 +0000 UTC m=+93.138937025" Jul 24 18:44:58.083698 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.083643 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-fvt2v" podStartSLOduration=1.41695149 podStartE2EDuration="12.083623918s" totalImagesPullingTime="10.666672428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:58.080682211 +0000 UTC m=+93.157194770" watchObservedRunningTime="2026-07-24 18:44:58.083623918 +0000 UTC m=+93.160136476" Jul 24 18:44:58.114061 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.114007 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-b7b9785b-csbzf" podStartSLOduration=2.12837641 podStartE2EDuration="13.113982781s" totalImagesPullingTime="10.985606371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:58.111835181 +0000 UTC m=+93.188347740" watchObservedRunningTime="2026-07-24 18:44:58.113982781 +0000 UTC m=+93.190495339" Jul 24 18:44:58.134463 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:44:58.134408 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7c7b8558b8-4d4ld" podStartSLOduration=2.199628973 podStartE2EDuration="13.134390993s" totalImagesPullingTime="10.93476202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:44:58.133173201 +0000 UTC m=+93.209685760" watchObservedRunningTime="2026-07-24 18:44:58.134390993 +0000 UTC m=+93.210903553" Jul 24 18:45:02.834923 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:02.834890 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x6l9k" Jul 24 18:45:06.534588 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.534549 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d9fd6769-k4ggz"] Jul 24 18:45:06.695124 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.695080 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8rwlb"] Jul 24 18:45:06.807507 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.807424 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.811088 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.811064 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 18:45:06.812373 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.812347 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l828c\"" Jul 24 18:45:06.812513 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.812393 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 18:45:06.812580 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.812535 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 18:45:06.812660 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.812642 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 18:45:06.855537 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.855512 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-textfile\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.855674 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.855545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-tls\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.855674 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.855570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.855674 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.855628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.855674 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.855655 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8872ad15-0b96-43dc-96a3-16e84a86af39-sys\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.855884 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.855675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8872ad15-0b96-43dc-96a3-16e84a86af39-root\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.855884 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.855699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-wtmp\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.855884 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.855715 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8872ad15-0b96-43dc-96a3-16e84a86af39-metrics-client-ca\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.855884 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.855778 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nkgk\" (UniqueName: \"kubernetes.io/projected/8872ad15-0b96-43dc-96a3-16e84a86af39-kube-api-access-2nkgk\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.956524 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.956490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-tls\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.956524 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.956524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.956777 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.956549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.956777 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.956682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8872ad15-0b96-43dc-96a3-16e84a86af39-sys\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.956777 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.956718 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8872ad15-0b96-43dc-96a3-16e84a86af39-root\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.956777 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.956761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-wtmp\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.956993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.956784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8872ad15-0b96-43dc-96a3-16e84a86af39-metrics-client-ca\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.956993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.956809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8872ad15-0b96-43dc-96a3-16e84a86af39-sys\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.956993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.956827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nkgk\" (UniqueName: \"kubernetes.io/projected/8872ad15-0b96-43dc-96a3-16e84a86af39-kube-api-access-2nkgk\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.956993 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.956949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-textfile\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.957192 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.957097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-wtmp\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.957192 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.957159 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8872ad15-0b96-43dc-96a3-16e84a86af39-root\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.957192 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.957177 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-accelerators-collector-config\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.957369 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.957224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-textfile\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.957834 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.957800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8872ad15-0b96-43dc-96a3-16e84a86af39-metrics-client-ca\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.959411 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.959371 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-tls\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.959503 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.959430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8872ad15-0b96-43dc-96a3-16e84a86af39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:06.965582 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:06.965561 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nkgk\" (UniqueName: \"kubernetes.io/projected/8872ad15-0b96-43dc-96a3-16e84a86af39-kube-api-access-2nkgk\") pod \"node-exporter-8rwlb\" (UID: \"8872ad15-0b96-43dc-96a3-16e84a86af39\") " pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:07.119263 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:07.119186 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8rwlb" Jul 24 18:45:08.514660 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:45:08.514626 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8872ad15_0b96_43dc_96a3_16e84a86af39.slice/crio-4358afee0a6624f7ebd0a2cc93e66db4b8eb94bd5857d8ae9cdad4b4bffc83cd WatchSource:0}: Error finding container 4358afee0a6624f7ebd0a2cc93e66db4b8eb94bd5857d8ae9cdad4b4bffc83cd: Status 404 returned error can't find the container with id 4358afee0a6624f7ebd0a2cc93e66db4b8eb94bd5857d8ae9cdad4b4bffc83cd Jul 24 18:45:09.069768 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:09.068423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-xbdq4" event={"ID":"88541e12-5e3b-47cd-947d-599b1501c178","Type":"ContainerStarted","Data":"98113e11d4d54ca411a39ef9fbd24b0d43526b1b29eb7b66793470d81fc228af"} Jul 24 18:45:09.070045 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:09.070017 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-xbdq4" Jul 24 18:45:09.072333 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:09.072302 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rwlb" event={"ID":"8872ad15-0b96-43dc-96a3-16e84a86af39","Type":"ContainerStarted","Data":"4358afee0a6624f7ebd0a2cc93e66db4b8eb94bd5857d8ae9cdad4b4bffc83cd"} Jul 24 18:45:09.074761 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:09.074708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d9fd6769-k4ggz" event={"ID":"65f99a57-f908-4e4c-a047-0e3d0b5fce74","Type":"ContainerStarted","Data":"874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563"} Jul 24 18:45:09.087467 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:09.087435 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-xbdq4" Jul 24 18:45:09.090234 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:09.090176 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-xbdq4" podStartSLOduration=2.08792479 podStartE2EDuration="24.090159713s" totalImagesPullingTime="22.002234923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:45:09.088903481 +0000 UTC m=+104.165416041" watchObservedRunningTime="2026-07-24 18:45:09.090159713 +0000 UTC m=+104.166672272" Jul 24 18:45:09.132270 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:09.132184 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8d9fd6769-k4ggz" podStartSLOduration=8.232376095 podStartE2EDuration="19.132163116s" totalImagesPullingTime="10.899787021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:45:09.132153644 +0000 UTC m=+104.208666202" watchObservedRunningTime="2026-07-24 18:45:09.132163116 +0000 UTC m=+104.208675675" Jul 24 18:45:10.373726 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:10.373695 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:45:11.087537 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:11.087495 2575 generic.go:356] "Generic (PLEG): container finished" podID="8872ad15-0b96-43dc-96a3-16e84a86af39" containerID="b466d8a6267edd13008d0e546ed5235f5caa8b6258c8c5b2a9321152a1866ec8" exitCode=0 Jul 24 18:45:11.087757 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:11.087598 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rwlb" event={"ID":"8872ad15-0b96-43dc-96a3-16e84a86af39","Type":"ContainerDied","Data":"b466d8a6267edd13008d0e546ed5235f5caa8b6258c8c5b2a9321152a1866ec8"} Jul 24 18:45:12.096526 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:12.096487 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rwlb" event={"ID":"8872ad15-0b96-43dc-96a3-16e84a86af39","Type":"ContainerStarted","Data":"6ba486e5ee09139c2753b5c32ffbee6f6f89d28a1d825072ed4adccc1c2b02a7"} Jul 24 18:45:12.096997 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:12.096565 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8rwlb" event={"ID":"8872ad15-0b96-43dc-96a3-16e84a86af39","Type":"ContainerStarted","Data":"a6cfca4215493b6922eff0bca87d0aa271c300ea59145dd6a3d50593803a4522"} Jul 24 18:45:12.118469 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:12.118416 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8rwlb" podStartSLOduration=4.250311578 podStartE2EDuration="6.118395063s" totalImagesPullingTime="1.765832084s" totalInitContainerRuntime="102.251401ms" isStatefulPod=true podCreationTimestamp="2026-07-24 18:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:45:12.115845434 +0000 UTC m=+107.192358183" watchObservedRunningTime="2026-07-24 18:45:12.118395063 +0000 UTC m=+107.194907621" Jul 24 18:45:12.484274 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:12.484237 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d954f5747-7kxdd"] Jul 24 18:45:23.152621 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:23.152588 2575 generic.go:356] "Generic (PLEG): container finished" podID="be74898d-3d60-437f-9c90-5408b5fbb3a3" containerID="895798cbf00458bd997008b9b179d08ec92761dde196233dd85ab728f46783b1" exitCode=0 Jul 24 18:45:23.153112 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:23.152663 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" event={"ID":"be74898d-3d60-437f-9c90-5408b5fbb3a3","Type":"ContainerDied","Data":"895798cbf00458bd997008b9b179d08ec92761dde196233dd85ab728f46783b1"} Jul 24 18:45:23.153112 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:23.153005 2575 scope.go:117] "RemoveContainer" containerID="895798cbf00458bd997008b9b179d08ec92761dde196233dd85ab728f46783b1" Jul 24 18:45:23.154235 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:23.154211 2575 generic.go:356] "Generic (PLEG): container finished" podID="4abb5614-6d9a-42f6-a655-cfbc38b39382" containerID="cef8659e4c56ebd36dbfed034854e00fd99e0db772e86a0b35d40ac53ef7b77c" exitCode=0 Jul 24 18:45:23.154310 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:23.154283 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" event={"ID":"4abb5614-6d9a-42f6-a655-cfbc38b39382","Type":"ContainerDied","Data":"cef8659e4c56ebd36dbfed034854e00fd99e0db772e86a0b35d40ac53ef7b77c"} Jul 24 18:45:23.154544 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:23.154531 2575 scope.go:117] "RemoveContainer" containerID="cef8659e4c56ebd36dbfed034854e00fd99e0db772e86a0b35d40ac53ef7b77c" Jul 24 18:45:24.160549 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:24.160514 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z" event={"ID":"4abb5614-6d9a-42f6-a655-cfbc38b39382","Type":"ContainerStarted","Data":"cc1ec2ce759c89d8f161ddad3352c7b003b53207a50abb21af9504112f7ff863"} Jul 24 18:45:24.162311 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:24.162286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mww9b" event={"ID":"be74898d-3d60-437f-9c90-5408b5fbb3a3","Type":"ContainerStarted","Data":"9c03a7e9fa11e91af5b402f5dc9c2f9d9e0011e370ba84dde26212feef011319"} Jul 24 18:45:34.106432 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.106376 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8d9fd6769-k4ggz" podUID="65f99a57-f908-4e4c-a047-0e3d0b5fce74" containerName="console" containerID="cri-o://874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563" gracePeriod=15 Jul 24 18:45:34.391882 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.391856 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:45:34.402302 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.402279 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-oauth-serving-cert\") pod \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " Jul 24 18:45:34.402427 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.402339 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbfcb\" (UniqueName: \"kubernetes.io/projected/65f99a57-f908-4e4c-a047-0e3d0b5fce74-kube-api-access-rbfcb\") pod \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " Jul 24 18:45:34.402427 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.402364 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-serving-cert\") pod \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " Jul 24 18:45:34.402427 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.402408 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-oauth-config\") pod \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " Jul 24 18:45:34.402584 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.402459 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-config\") pod \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " Jul 24 18:45:34.402584 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.402498 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-service-ca\") pod \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\" (UID: \"65f99a57-f908-4e4c-a047-0e3d0b5fce74\") " Jul 24 18:45:34.403023 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.402990 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "65f99a57-f908-4e4c-a047-0e3d0b5fce74" (UID: "65f99a57-f908-4e4c-a047-0e3d0b5fce74"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:45:34.403239 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.403206 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-config" (OuterVolumeSpecName: "console-config") pod "65f99a57-f908-4e4c-a047-0e3d0b5fce74" (UID: "65f99a57-f908-4e4c-a047-0e3d0b5fce74"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:45:34.403671 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.403647 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-service-ca" (OuterVolumeSpecName: "service-ca") pod "65f99a57-f908-4e4c-a047-0e3d0b5fce74" (UID: "65f99a57-f908-4e4c-a047-0e3d0b5fce74"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:45:34.408788 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.407191 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f99a57-f908-4e4c-a047-0e3d0b5fce74-kube-api-access-rbfcb" (OuterVolumeSpecName: "kube-api-access-rbfcb") pod "65f99a57-f908-4e4c-a047-0e3d0b5fce74" (UID: "65f99a57-f908-4e4c-a047-0e3d0b5fce74"). InnerVolumeSpecName "kube-api-access-rbfcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:45:34.408788 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.407405 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "65f99a57-f908-4e4c-a047-0e3d0b5fce74" (UID: "65f99a57-f908-4e4c-a047-0e3d0b5fce74"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:45:34.409838 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.409797 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "65f99a57-f908-4e4c-a047-0e3d0b5fce74" (UID: "65f99a57-f908-4e4c-a047-0e3d0b5fce74"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:45:34.504075 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.504040 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:34.504237 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.504091 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-oauth-serving-cert\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:34.504237 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.504102 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbfcb\" (UniqueName: \"kubernetes.io/projected/65f99a57-f908-4e4c-a047-0e3d0b5fce74-kube-api-access-rbfcb\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:34.504237 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.504112 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-serving-cert\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:34.504237 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.504120 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-oauth-config\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:34.504237 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:34.504128 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65f99a57-f908-4e4c-a047-0e3d0b5fce74-console-config\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:35.207788 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:35.207722 2575 generic.go:356] "Generic (PLEG): container finished" podID="65f99a57-f908-4e4c-a047-0e3d0b5fce74" containerID="874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563" exitCode=0 Jul 24 18:45:35.208292 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:35.207812 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d9fd6769-k4ggz" Jul 24 18:45:35.208292 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:35.207810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d9fd6769-k4ggz" event={"ID":"65f99a57-f908-4e4c-a047-0e3d0b5fce74","Type":"ContainerDied","Data":"874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563"} Jul 24 18:45:35.208292 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:35.207933 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d9fd6769-k4ggz" event={"ID":"65f99a57-f908-4e4c-a047-0e3d0b5fce74","Type":"ContainerDied","Data":"82ea93e55d198c216fd77f2d859f6815503b3bcaf23d4f018451ff74eafb3021"} Jul 24 18:45:35.208292 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:35.207956 2575 scope.go:117] "RemoveContainer" containerID="874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563" Jul 24 18:45:35.216906 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:35.216886 2575 scope.go:117] "RemoveContainer" containerID="874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563" Jul 24 18:45:35.217155 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:45:35.217137 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563\": container with ID starting with 874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563 not found: ID does not exist" containerID="874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563" Jul 24 18:45:35.217209 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:35.217161 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563"} err="failed to get container status \"874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563\": rpc error: code = NotFound desc = could not find container \"874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563\": container with ID starting with 874787ff6456e0c217f51a865d93296f0281e2f43508ef6fbddc884d15407563 not found: ID does not exist" Jul 24 18:45:35.241308 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:35.241278 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d9fd6769-k4ggz"] Jul 24 18:45:35.246077 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:35.246057 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8d9fd6769-k4ggz"] Jul 24 18:45:35.415556 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:35.415527 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f99a57-f908-4e4c-a047-0e3d0b5fce74" path="/var/lib/kubelet/pods/65f99a57-f908-4e4c-a047-0e3d0b5fce74/volumes" Jul 24 18:45:37.508144 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.508097 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" podUID="425fa486-2440-4be2-b6fb-34f609ab97d5" containerName="registry" containerID="cri-o://2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691" gracePeriod=30 Jul 24 18:45:37.760206 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.760147 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:45:37.829734 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.829703 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-trusted-ca\") pod \"425fa486-2440-4be2-b6fb-34f609ab97d5\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " Jul 24 18:45:37.829891 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.829755 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-bound-sa-token\") pod \"425fa486-2440-4be2-b6fb-34f609ab97d5\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " Jul 24 18:45:37.829891 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.829793 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7wln\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-kube-api-access-j7wln\") pod \"425fa486-2440-4be2-b6fb-34f609ab97d5\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " Jul 24 18:45:37.829991 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.829968 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/425fa486-2440-4be2-b6fb-34f609ab97d5-ca-trust-extracted\") pod \"425fa486-2440-4be2-b6fb-34f609ab97d5\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " Jul 24 18:45:37.830038 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.830023 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-image-registry-private-configuration\") pod \"425fa486-2440-4be2-b6fb-34f609ab97d5\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " Jul 24 18:45:37.830095 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.830080 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-installation-pull-secrets\") pod \"425fa486-2440-4be2-b6fb-34f609ab97d5\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " Jul 24 18:45:37.830137 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.830110 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") pod \"425fa486-2440-4be2-b6fb-34f609ab97d5\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " Jul 24 18:45:37.830182 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.830142 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-certificates\") pod \"425fa486-2440-4be2-b6fb-34f609ab97d5\" (UID: \"425fa486-2440-4be2-b6fb-34f609ab97d5\") " Jul 24 18:45:37.830440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.830244 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "425fa486-2440-4be2-b6fb-34f609ab97d5" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:45:37.830440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.830390 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-trusted-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:37.831054 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.831017 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "425fa486-2440-4be2-b6fb-34f609ab97d5" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:45:37.833406 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.833371 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "425fa486-2440-4be2-b6fb-34f609ab97d5" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:45:37.833507 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.833419 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "425fa486-2440-4be2-b6fb-34f609ab97d5" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:45:37.833507 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.833485 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "425fa486-2440-4be2-b6fb-34f609ab97d5" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:45:37.833623 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.833524 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "425fa486-2440-4be2-b6fb-34f609ab97d5" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:45:37.833623 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.833604 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-kube-api-access-j7wln" (OuterVolumeSpecName: "kube-api-access-j7wln") pod "425fa486-2440-4be2-b6fb-34f609ab97d5" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5"). InnerVolumeSpecName "kube-api-access-j7wln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:45:37.838704 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.838679 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425fa486-2440-4be2-b6fb-34f609ab97d5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "425fa486-2440-4be2-b6fb-34f609ab97d5" (UID: "425fa486-2440-4be2-b6fb-34f609ab97d5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:45:37.931385 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.931351 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-bound-sa-token\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:37.931385 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.931381 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7wln\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-kube-api-access-j7wln\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:37.931385 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.931390 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/425fa486-2440-4be2-b6fb-34f609ab97d5-ca-trust-extracted\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:37.931588 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.931400 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-image-registry-private-configuration\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:37.931588 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.931411 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/425fa486-2440-4be2-b6fb-34f609ab97d5-installation-pull-secrets\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:37.931588 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.931421 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-tls\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:37.931588 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:37.931431 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/425fa486-2440-4be2-b6fb-34f609ab97d5-registry-certificates\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:45:38.221845 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:38.221814 2575 generic.go:356] "Generic (PLEG): container finished" podID="425fa486-2440-4be2-b6fb-34f609ab97d5" containerID="2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691" exitCode=0 Jul 24 18:45:38.221968 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:38.221870 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" Jul 24 18:45:38.221968 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:38.221898 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" event={"ID":"425fa486-2440-4be2-b6fb-34f609ab97d5","Type":"ContainerDied","Data":"2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691"} Jul 24 18:45:38.222071 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:38.221973 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d954f5747-7kxdd" event={"ID":"425fa486-2440-4be2-b6fb-34f609ab97d5","Type":"ContainerDied","Data":"c30e7c388c4c859c2d75f084aa3697b8a4b02144d635daa28d11eebd95531119"} Jul 24 18:45:38.222071 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:38.221994 2575 scope.go:117] "RemoveContainer" containerID="2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691" Jul 24 18:45:38.259183 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:38.259159 2575 scope.go:117] "RemoveContainer" containerID="2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691" Jul 24 18:45:38.259605 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:45:38.259582 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691\": container with ID starting with 2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691 not found: ID does not exist" containerID="2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691" Jul 24 18:45:38.259709 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:38.259613 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691"} err="failed to get container status \"2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691\": rpc error: code = NotFound desc = could not find container \"2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691\": container with ID starting with 2165d836ab9d2fe4c4872f2bae527c65d40982b4bfdc38550f4cbce6ecc86691 not found: ID does not exist" Jul 24 18:45:38.269700 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:38.269677 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d954f5747-7kxdd"] Jul 24 18:45:38.275464 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:38.275442 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d954f5747-7kxdd"] Jul 24 18:45:39.228874 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:39.228843 2575 generic.go:356] "Generic (PLEG): container finished" podID="5fc61054-0130-44ec-a896-11d6124b50d6" containerID="03f7e231f928f2a9b2ae79d77a623fb2901fadfc56844e707e08493cda0069b5" exitCode=0 Jul 24 18:45:39.229283 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:39.228882 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-6cl5p" event={"ID":"5fc61054-0130-44ec-a896-11d6124b50d6","Type":"ContainerDied","Data":"03f7e231f928f2a9b2ae79d77a623fb2901fadfc56844e707e08493cda0069b5"} Jul 24 18:45:39.229283 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:39.229230 2575 scope.go:117] "RemoveContainer" containerID="03f7e231f928f2a9b2ae79d77a623fb2901fadfc56844e707e08493cda0069b5" Jul 24 18:45:39.417272 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:39.417234 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="425fa486-2440-4be2-b6fb-34f609ab97d5" path="/var/lib/kubelet/pods/425fa486-2440-4be2-b6fb-34f609ab97d5/volumes" Jul 24 18:45:40.234720 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:45:40.234685 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-6cl5p" event={"ID":"5fc61054-0130-44ec-a896-11d6124b50d6","Type":"ContainerStarted","Data":"5c4da184f91d826788d041a97c461922fd368b55c4568a2945d5b40cc93f2240"} Jul 24 18:48:25.344214 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:25.344183 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 18:48:25.351619 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:25.351594 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 18:48:25.354140 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:25.354120 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 18:48:25.355323 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:25.355304 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 18:48:25.362414 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:25.362394 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 18:48:29.694408 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.694371 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8"] Jul 24 18:48:29.696759 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.694660 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425fa486-2440-4be2-b6fb-34f609ab97d5" containerName="registry" Jul 24 18:48:29.696759 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.694671 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="425fa486-2440-4be2-b6fb-34f609ab97d5" containerName="registry" Jul 24 18:48:29.696759 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.694682 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f99a57-f908-4e4c-a047-0e3d0b5fce74" containerName="console" Jul 24 18:48:29.696759 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.694688 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f99a57-f908-4e4c-a047-0e3d0b5fce74" containerName="console" Jul 24 18:48:29.696759 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.694767 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f99a57-f908-4e4c-a047-0e3d0b5fce74" containerName="console" Jul 24 18:48:29.696759 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.694777 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="425fa486-2440-4be2-b6fb-34f609ab97d5" containerName="registry" Jul 24 18:48:29.697657 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.697642 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.701308 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.701287 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 18:48:29.701448 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.701428 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 18:48:29.702959 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.702926 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 18:48:29.703609 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.703592 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 18:48:29.704651 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.704635 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7rgj8\"" Jul 24 18:48:29.800177 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.800141 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8"] Jul 24 18:48:29.805635 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.805609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-tmp\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.805820 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.805639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.805820 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.805661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2jr\" (UniqueName: \"kubernetes.io/projected/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-kube-api-access-6m2jr\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.805820 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.805719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-manifests\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.805820 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.805794 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-webhook-cert\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.906604 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.906568 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-tmp\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.906834 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.906610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.906834 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.906638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2jr\" (UniqueName: \"kubernetes.io/projected/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-kube-api-access-6m2jr\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.906834 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.906670 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-manifests\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.906834 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.906708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-webhook-cert\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.907176 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.907145 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-manifests\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.909229 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.909200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-tmp\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.909387 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.909369 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-webhook-cert\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.909448 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.909372 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:29.926124 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:29.926099 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2jr\" (UniqueName: \"kubernetes.io/projected/bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe-kube-api-access-6m2jr\") pod \"opendatahub-operator-controller-manager-6559d49d8b-fmds8\" (UID: \"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe\") " pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:30.007661 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:30.007578 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:30.160260 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:30.160234 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8"] Jul 24 18:48:30.163278 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:48:30.163251 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd4d9e6_ba36_478f_9171_7cffa5bd2dfe.slice/crio-18340e2386861ae7c28c3f55814a6d1c46dc07fc36b6f5e4eb2389f94d2cfe57 WatchSource:0}: Error finding container 18340e2386861ae7c28c3f55814a6d1c46dc07fc36b6f5e4eb2389f94d2cfe57: Status 404 returned error can't find the container with id 18340e2386861ae7c28c3f55814a6d1c46dc07fc36b6f5e4eb2389f94d2cfe57 Jul 24 18:48:30.164951 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:30.164934 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 18:48:30.911950 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:30.911843 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" event={"ID":"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe","Type":"ContainerStarted","Data":"18340e2386861ae7c28c3f55814a6d1c46dc07fc36b6f5e4eb2389f94d2cfe57"} Jul 24 18:48:33.927810 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:33.927775 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" event={"ID":"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe","Type":"ContainerStarted","Data":"836f93eb2dae1e65973bd3aeadc8cf7814a2d287383f43b5c6d7e82cbad31bb6"} Jul 24 18:48:34.933503 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:34.933472 2575 generic.go:356] "Generic (PLEG): container finished" podID="bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe" containerID="836f93eb2dae1e65973bd3aeadc8cf7814a2d287383f43b5c6d7e82cbad31bb6" exitCode=0 Jul 24 18:48:34.933968 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:34.933547 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" event={"ID":"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe","Type":"ContainerDied","Data":"836f93eb2dae1e65973bd3aeadc8cf7814a2d287383f43b5c6d7e82cbad31bb6"} Jul 24 18:48:35.939882 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:35.939847 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" event={"ID":"bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe","Type":"ContainerStarted","Data":"d98d634c65197dd2fdfc916c980d31f58be7fbe6fa14f2c32a1bd78879100163"} Jul 24 18:48:35.940274 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:35.939993 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:35.965062 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:35.965015 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" podStartSLOduration=2.64501714 podStartE2EDuration="6.96499909s" totalImagesPullingTime="3.301155058s" totalInitContainerRuntime="1.018826892s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:48:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:48:35.962623924 +0000 UTC m=+311.039136547" watchObservedRunningTime="2026-07-24 18:48:35.96499909 +0000 UTC m=+311.041511648" Jul 24 18:48:46.947306 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:46.947277 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6559d49d8b-fmds8" Jul 24 18:48:49.215323 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.215291 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst"] Jul 24 18:48:49.220705 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.220686 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.225656 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.225634 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-z2vvb\"" Jul 24 18:48:49.225800 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.225722 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 18:48:49.225893 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.225876 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 18:48:49.242557 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.242528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pkrd\" (UniqueName: \"kubernetes.io/projected/e6e3085d-cade-4c35-9802-156cbcccaad5-kube-api-access-6pkrd\") pod \"kube-auth-proxy-7d65fbd68-jmdst\" (UID: \"e6e3085d-cade-4c35-9802-156cbcccaad5\") " pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.242650 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.242561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6e3085d-cade-4c35-9802-156cbcccaad5-tmp\") pod \"kube-auth-proxy-7d65fbd68-jmdst\" (UID: \"e6e3085d-cade-4c35-9802-156cbcccaad5\") " pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.242650 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.242639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e3085d-cade-4c35-9802-156cbcccaad5-tls-certs\") pod \"kube-auth-proxy-7d65fbd68-jmdst\" (UID: \"e6e3085d-cade-4c35-9802-156cbcccaad5\") " pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.251553 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.251533 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst"] Jul 24 18:48:49.348287 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.348238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e3085d-cade-4c35-9802-156cbcccaad5-tls-certs\") pod \"kube-auth-proxy-7d65fbd68-jmdst\" (UID: \"e6e3085d-cade-4c35-9802-156cbcccaad5\") " pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.348695 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.348663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkrd\" (UniqueName: \"kubernetes.io/projected/e6e3085d-cade-4c35-9802-156cbcccaad5-kube-api-access-6pkrd\") pod \"kube-auth-proxy-7d65fbd68-jmdst\" (UID: \"e6e3085d-cade-4c35-9802-156cbcccaad5\") " pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.348867 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.348713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6e3085d-cade-4c35-9802-156cbcccaad5-tmp\") pod \"kube-auth-proxy-7d65fbd68-jmdst\" (UID: \"e6e3085d-cade-4c35-9802-156cbcccaad5\") " pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.351013 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.350987 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6e3085d-cade-4c35-9802-156cbcccaad5-tmp\") pod \"kube-auth-proxy-7d65fbd68-jmdst\" (UID: \"e6e3085d-cade-4c35-9802-156cbcccaad5\") " pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.351287 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.351256 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e3085d-cade-4c35-9802-156cbcccaad5-tls-certs\") pod \"kube-auth-proxy-7d65fbd68-jmdst\" (UID: \"e6e3085d-cade-4c35-9802-156cbcccaad5\") " pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.360714 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.360694 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkrd\" (UniqueName: \"kubernetes.io/projected/e6e3085d-cade-4c35-9802-156cbcccaad5-kube-api-access-6pkrd\") pod \"kube-auth-proxy-7d65fbd68-jmdst\" (UID: \"e6e3085d-cade-4c35-9802-156cbcccaad5\") " pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.416377 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.416347 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-45v4h"] Jul 24 18:48:49.419303 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.419282 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:49.423513 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.423488 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 18:48:49.423622 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.423581 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-t2675\"" Jul 24 18:48:49.442265 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.442238 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-45v4h"] Jul 24 18:48:49.449942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.449920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/238e37c5-ac35-446e-80c5-4fd5ba500e2a-cert\") pod \"odh-model-controller-7998c67dc4-45v4h\" (UID: \"238e37c5-ac35-446e-80c5-4fd5ba500e2a\") " pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:49.450130 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.450077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwf6\" (UniqueName: \"kubernetes.io/projected/238e37c5-ac35-446e-80c5-4fd5ba500e2a-kube-api-access-mlwf6\") pod \"odh-model-controller-7998c67dc4-45v4h\" (UID: \"238e37c5-ac35-446e-80c5-4fd5ba500e2a\") " pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:49.530799 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.530701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" Jul 24 18:48:49.550692 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.550665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/238e37c5-ac35-446e-80c5-4fd5ba500e2a-cert\") pod \"odh-model-controller-7998c67dc4-45v4h\" (UID: \"238e37c5-ac35-446e-80c5-4fd5ba500e2a\") " pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:49.550832 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.550777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwf6\" (UniqueName: \"kubernetes.io/projected/238e37c5-ac35-446e-80c5-4fd5ba500e2a-kube-api-access-mlwf6\") pod \"odh-model-controller-7998c67dc4-45v4h\" (UID: \"238e37c5-ac35-446e-80c5-4fd5ba500e2a\") " pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:49.550898 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:48:49.550872 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 24 18:48:49.550951 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:48:49.550943 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/238e37c5-ac35-446e-80c5-4fd5ba500e2a-cert podName:238e37c5-ac35-446e-80c5-4fd5ba500e2a nodeName:}" failed. No retries permitted until 2026-07-24 18:48:50.050920937 +0000 UTC m=+325.127433482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/238e37c5-ac35-446e-80c5-4fd5ba500e2a-cert") pod "odh-model-controller-7998c67dc4-45v4h" (UID: "238e37c5-ac35-446e-80c5-4fd5ba500e2a") : secret "odh-model-controller-webhook-cert" not found Jul 24 18:48:49.570156 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.570123 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwf6\" (UniqueName: \"kubernetes.io/projected/238e37c5-ac35-446e-80c5-4fd5ba500e2a-kube-api-access-mlwf6\") pod \"odh-model-controller-7998c67dc4-45v4h\" (UID: \"238e37c5-ac35-446e-80c5-4fd5ba500e2a\") " pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:49.678233 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:49.678140 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst"] Jul 24 18:48:49.680926 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:48:49.680900 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e3085d_cade_4c35_9802_156cbcccaad5.slice/crio-bf30b4865663b3c17d528c2bf3a9d1aae57f032fc63269ed922ac901fd4c5015 WatchSource:0}: Error finding container bf30b4865663b3c17d528c2bf3a9d1aae57f032fc63269ed922ac901fd4c5015: Status 404 returned error can't find the container with id bf30b4865663b3c17d528c2bf3a9d1aae57f032fc63269ed922ac901fd4c5015 Jul 24 18:48:50.000637 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.000592 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" event={"ID":"e6e3085d-cade-4c35-9802-156cbcccaad5","Type":"ContainerStarted","Data":"bf30b4865663b3c17d528c2bf3a9d1aae57f032fc63269ed922ac901fd4c5015"} Jul 24 18:48:50.054980 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.054945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/238e37c5-ac35-446e-80c5-4fd5ba500e2a-cert\") pod \"odh-model-controller-7998c67dc4-45v4h\" (UID: \"238e37c5-ac35-446e-80c5-4fd5ba500e2a\") " pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:50.055137 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:48:50.055064 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 24 18:48:50.055137 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:48:50.055122 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/238e37c5-ac35-446e-80c5-4fd5ba500e2a-cert podName:238e37c5-ac35-446e-80c5-4fd5ba500e2a nodeName:}" failed. No retries permitted until 2026-07-24 18:48:51.055103565 +0000 UTC m=+326.131616101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/238e37c5-ac35-446e-80c5-4fd5ba500e2a-cert") pod "odh-model-controller-7998c67dc4-45v4h" (UID: "238e37c5-ac35-446e-80c5-4fd5ba500e2a") : secret "odh-model-controller-webhook-cert" not found Jul 24 18:48:50.676051 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.676015 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cw2kq"] Jul 24 18:48:50.679447 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.679427 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:48:50.683042 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.683010 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 24 18:48:50.683160 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.683091 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-xq7b9\"" Jul 24 18:48:50.697172 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.697145 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cw2kq"] Jul 24 18:48:50.760939 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.760908 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq2tz\" (UniqueName: \"kubernetes.io/projected/a96b24b8-3be4-4d40-86ab-4041b8edd223-kube-api-access-wq2tz\") pod \"model-serving-api-54d89bd5fc-cw2kq\" (UID: \"a96b24b8-3be4-4d40-86ab-4041b8edd223\") " pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:48:50.761080 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.760946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96b24b8-3be4-4d40-86ab-4041b8edd223-tls-certs\") pod \"model-serving-api-54d89bd5fc-cw2kq\" (UID: \"a96b24b8-3be4-4d40-86ab-4041b8edd223\") " pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:48:50.810307 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.810272 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb"] Jul 24 18:48:50.813839 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.813822 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.818560 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.818534 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 18:48:50.818675 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.818605 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hfrkk\"" Jul 24 18:48:50.819218 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.819193 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 18:48:50.820234 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.820213 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 18:48:50.821367 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.821350 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 18:48:50.844387 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.844365 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb"] Jul 24 18:48:50.861809 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.861785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.861932 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.861822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.861932 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.861884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.861932 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.861905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pqqp\" (UniqueName: \"kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-kube-api-access-5pqqp\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.861932 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.861926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4dd75b-b08b-4653-9059-226877783e11-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.862127 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.861939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.862127 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.861991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq2tz\" (UniqueName: \"kubernetes.io/projected/a96b24b8-3be4-4d40-86ab-4041b8edd223-kube-api-access-wq2tz\") pod \"model-serving-api-54d89bd5fc-cw2kq\" (UID: \"a96b24b8-3be4-4d40-86ab-4041b8edd223\") " pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:48:50.862127 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.862020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96b24b8-3be4-4d40-86ab-4041b8edd223-tls-certs\") pod \"model-serving-api-54d89bd5fc-cw2kq\" (UID: \"a96b24b8-3be4-4d40-86ab-4041b8edd223\") " pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:48:50.862127 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.862036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.862350 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:48:50.862331 2575 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 24 18:48:50.862462 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:48:50.862407 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a96b24b8-3be4-4d40-86ab-4041b8edd223-tls-certs podName:a96b24b8-3be4-4d40-86ab-4041b8edd223 nodeName:}" failed. No retries permitted until 2026-07-24 18:48:51.362385847 +0000 UTC m=+326.438898391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a96b24b8-3be4-4d40-86ab-4041b8edd223-tls-certs") pod "model-serving-api-54d89bd5fc-cw2kq" (UID: "a96b24b8-3be4-4d40-86ab-4041b8edd223") : secret "model-serving-api-tls" not found Jul 24 18:48:50.884724 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.884696 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq2tz\" (UniqueName: \"kubernetes.io/projected/a96b24b8-3be4-4d40-86ab-4041b8edd223-kube-api-access-wq2tz\") pod \"model-serving-api-54d89bd5fc-cw2kq\" (UID: \"a96b24b8-3be4-4d40-86ab-4041b8edd223\") " pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:48:50.962905 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.962831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.962905 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.962871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pqqp\" (UniqueName: \"kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-kube-api-access-5pqqp\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.963100 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.962914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4dd75b-b08b-4653-9059-226877783e11-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.963100 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.962937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.963100 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.963007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.963100 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.963037 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.963100 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.963074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.963865 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.963833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.965459 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.965438 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.965888 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.965867 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.965971 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.965873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4dd75b-b08b-4653-9059-226877783e11-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.966020 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.965979 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.974362 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.974336 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:50.975009 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:50.974991 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pqqp\" (UniqueName: \"kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-kube-api-access-5pqqp\") pod \"istiod-openshift-gateway-7b7995d5cd-qtzgb\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:51.064455 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:51.064423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/238e37c5-ac35-446e-80c5-4fd5ba500e2a-cert\") pod \"odh-model-controller-7998c67dc4-45v4h\" (UID: \"238e37c5-ac35-446e-80c5-4fd5ba500e2a\") " pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:51.066889 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:51.066872 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/238e37c5-ac35-446e-80c5-4fd5ba500e2a-cert\") pod \"odh-model-controller-7998c67dc4-45v4h\" (UID: \"238e37c5-ac35-446e-80c5-4fd5ba500e2a\") " pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:51.123411 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:51.123380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:51.229052 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:51.229008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:51.284232 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:51.284202 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb"] Jul 24 18:48:51.288525 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:48:51.288038 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4dd75b_b08b_4653_9059_226877783e11.slice/crio-f40c5dc2b32a3f6918ced2e2eed6f901d99b7a389d5b90257f47e75dbf8c0c8a WatchSource:0}: Error finding container f40c5dc2b32a3f6918ced2e2eed6f901d99b7a389d5b90257f47e75dbf8c0c8a: Status 404 returned error can't find the container with id f40c5dc2b32a3f6918ced2e2eed6f901d99b7a389d5b90257f47e75dbf8c0c8a Jul 24 18:48:51.366692 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:51.366659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96b24b8-3be4-4d40-86ab-4041b8edd223-tls-certs\") pod \"model-serving-api-54d89bd5fc-cw2kq\" (UID: \"a96b24b8-3be4-4d40-86ab-4041b8edd223\") " pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:48:51.369645 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:51.369608 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a96b24b8-3be4-4d40-86ab-4041b8edd223-tls-certs\") pod \"model-serving-api-54d89bd5fc-cw2kq\" (UID: \"a96b24b8-3be4-4d40-86ab-4041b8edd223\") " pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:48:51.385140 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:51.385108 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-45v4h"] Jul 24 18:48:51.387526 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:48:51.387487 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238e37c5_ac35_446e_80c5_4fd5ba500e2a.slice/crio-ec65a145208b3f851e2a5869ed6b290ffa4a36b3223f4721020c8ff94f4744c4 WatchSource:0}: Error finding container ec65a145208b3f851e2a5869ed6b290ffa4a36b3223f4721020c8ff94f4744c4: Status 404 returned error can't find the container with id ec65a145208b3f851e2a5869ed6b290ffa4a36b3223f4721020c8ff94f4744c4 Jul 24 18:48:51.594065 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:51.593975 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:48:51.736245 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:51.736128 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-cw2kq"] Jul 24 18:48:51.738716 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:48:51.738685 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96b24b8_3be4_4d40_86ab_4041b8edd223.slice/crio-cd48e8bb71ee21c680fdbfb06f390964371f2325b563534908f940429b539126 WatchSource:0}: Error finding container cd48e8bb71ee21c680fdbfb06f390964371f2325b563534908f940429b539126: Status 404 returned error can't find the container with id cd48e8bb71ee21c680fdbfb06f390964371f2325b563534908f940429b539126 Jul 24 18:48:52.014983 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:52.014946 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" event={"ID":"a96b24b8-3be4-4d40-86ab-4041b8edd223","Type":"ContainerStarted","Data":"cd48e8bb71ee21c680fdbfb06f390964371f2325b563534908f940429b539126"} Jul 24 18:48:52.016481 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:52.016416 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" event={"ID":"238e37c5-ac35-446e-80c5-4fd5ba500e2a","Type":"ContainerStarted","Data":"ec65a145208b3f851e2a5869ed6b290ffa4a36b3223f4721020c8ff94f4744c4"} Jul 24 18:48:52.017858 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:52.017824 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" event={"ID":"3e4dd75b-b08b-4653-9059-226877783e11","Type":"ContainerStarted","Data":"f40c5dc2b32a3f6918ced2e2eed6f901d99b7a389d5b90257f47e75dbf8c0c8a"} Jul 24 18:48:53.027328 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:53.027245 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" event={"ID":"e6e3085d-cade-4c35-9802-156cbcccaad5","Type":"ContainerStarted","Data":"44ba395be7c2527b21d8340d8b9b6748fbc09e74cee6ae999a4ca224bf7f6441"} Jul 24 18:48:53.050734 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:53.050669 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d65fbd68-jmdst" podStartSLOduration=1.080791872 podStartE2EDuration="4.05064916s" totalImagesPullingTime="2.969857288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:48:53.045426629 +0000 UTC m=+328.121939188" watchObservedRunningTime="2026-07-24 18:48:53.05064916 +0000 UTC m=+328.127161719" Jul 24 18:48:55.039470 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:55.039431 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" event={"ID":"a96b24b8-3be4-4d40-86ab-4041b8edd223","Type":"ContainerStarted","Data":"5b0ce2748952b5046e66a70a73d73bf66a9ec37415634592ffd9700728d1ed54"} Jul 24 18:48:55.039931 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:55.039563 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:48:55.041038 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:55.041014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" event={"ID":"238e37c5-ac35-446e-80c5-4fd5ba500e2a","Type":"ContainerStarted","Data":"05e7357dda8847db5676661187645809ea97480357ba3cd813536fb5a978494b"} Jul 24 18:48:55.041169 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:55.041155 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:48:55.058862 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:55.058814 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" podStartSLOduration=2.266576895 podStartE2EDuration="5.058799228s" totalImagesPullingTime="2.792222333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:48:55.057361883 +0000 UTC m=+330.133874441" watchObservedRunningTime="2026-07-24 18:48:55.058799228 +0000 UTC m=+330.135311785" Jul 24 18:48:55.074187 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:55.074139 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" podStartSLOduration=2.9280606909999998 podStartE2EDuration="6.074124048s" totalImagesPullingTime="3.146063357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:48:55.07361599 +0000 UTC m=+330.150128548" watchObservedRunningTime="2026-07-24 18:48:55.074124048 +0000 UTC m=+330.150636606" Jul 24 18:48:57.180323 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:57.180283 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:48:57.180633 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:57.180354 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:48:58.066720 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:58.066625 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" event={"ID":"3e4dd75b-b08b-4653-9059-226877783e11","Type":"ContainerStarted","Data":"2656bd1434d412a47c58b25cc2f0d75c0762833274eff6f3a2979be5f7faa559"} Jul 24 18:48:58.067599 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:58.067565 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:58.069437 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:58.069410 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-qtzgb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 18:48:58.069569 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:58.069467 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" podUID="3e4dd75b-b08b-4653-9059-226877783e11" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 18:48:59.072002 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:59.071971 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:48:59.107539 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:48:59.107463 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" podStartSLOduration=3.218988065 podStartE2EDuration="9.107439995s" totalImagesPullingTime="5.88845193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:48:58.113111232 +0000 UTC m=+333.189623791" watchObservedRunningTime="2026-07-24 18:48:59.107439995 +0000 UTC m=+334.183952554" Jul 24 18:49:00.500556 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:00.500527 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8"] Jul 24 18:49:00.503720 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:00.503701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:00.509632 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:00.509611 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2d2cm\"" Jul 24 18:49:00.510995 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:00.510971 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 18:49:00.555143 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:00.555112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8n2\" (UniqueName: \"kubernetes.io/projected/421e836f-94c8-469e-9af3-8b8732d6282d-kube-api-access-fx8n2\") pod \"kserve-controller-manager-d6f4b65b8-7gzl8\" (UID: \"421e836f-94c8-469e-9af3-8b8732d6282d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:00.555323 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:00.555164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/421e836f-94c8-469e-9af3-8b8732d6282d-cert\") pod \"kserve-controller-manager-d6f4b65b8-7gzl8\" (UID: \"421e836f-94c8-469e-9af3-8b8732d6282d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:00.634576 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:00.634543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8"] Jul 24 18:49:00.656588 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:00.656557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8n2\" (UniqueName: \"kubernetes.io/projected/421e836f-94c8-469e-9af3-8b8732d6282d-kube-api-access-fx8n2\") pod \"kserve-controller-manager-d6f4b65b8-7gzl8\" (UID: \"421e836f-94c8-469e-9af3-8b8732d6282d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:00.656772 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:00.656597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/421e836f-94c8-469e-9af3-8b8732d6282d-cert\") pod \"kserve-controller-manager-d6f4b65b8-7gzl8\" (UID: \"421e836f-94c8-469e-9af3-8b8732d6282d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:00.656772 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:49:00.656727 2575 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 24 18:49:00.656894 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:49:00.656827 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/421e836f-94c8-469e-9af3-8b8732d6282d-cert podName:421e836f-94c8-469e-9af3-8b8732d6282d nodeName:}" failed. No retries permitted until 2026-07-24 18:49:01.156806999 +0000 UTC m=+336.233319534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/421e836f-94c8-469e-9af3-8b8732d6282d-cert") pod "kserve-controller-manager-d6f4b65b8-7gzl8" (UID: "421e836f-94c8-469e-9af3-8b8732d6282d") : secret "kserve-webhook-server-cert" not found Jul 24 18:49:00.680948 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:00.680922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8n2\" (UniqueName: \"kubernetes.io/projected/421e836f-94c8-469e-9af3-8b8732d6282d-kube-api-access-fx8n2\") pod \"kserve-controller-manager-d6f4b65b8-7gzl8\" (UID: \"421e836f-94c8-469e-9af3-8b8732d6282d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:01.161779 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:01.161721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/421e836f-94c8-469e-9af3-8b8732d6282d-cert\") pod \"kserve-controller-manager-d6f4b65b8-7gzl8\" (UID: \"421e836f-94c8-469e-9af3-8b8732d6282d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:01.164294 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:01.164272 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/421e836f-94c8-469e-9af3-8b8732d6282d-cert\") pod \"kserve-controller-manager-d6f4b65b8-7gzl8\" (UID: \"421e836f-94c8-469e-9af3-8b8732d6282d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:01.414168 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:01.414081 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:01.573380 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:01.573357 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8"] Jul 24 18:49:01.575949 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:49:01.575922 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421e836f_94c8_469e_9af3_8b8732d6282d.slice/crio-0df1d5ea066a1689c2fa91faa28ad1915e1b9eef6cfcd408c7754a4c1c283050 WatchSource:0}: Error finding container 0df1d5ea066a1689c2fa91faa28ad1915e1b9eef6cfcd408c7754a4c1c283050: Status 404 returned error can't find the container with id 0df1d5ea066a1689c2fa91faa28ad1915e1b9eef6cfcd408c7754a4c1c283050 Jul 24 18:49:02.085659 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:02.085623 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" event={"ID":"421e836f-94c8-469e-9af3-8b8732d6282d","Type":"ContainerStarted","Data":"0df1d5ea066a1689c2fa91faa28ad1915e1b9eef6cfcd408c7754a4c1c283050"} Jul 24 18:49:03.454375 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:03.454341 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb"] Jul 24 18:49:03.454793 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:03.454505 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" podUID="3e4dd75b-b08b-4653-9059-226877783e11" containerName="discovery" containerID="cri-o://2656bd1434d412a47c58b25cc2f0d75c0762833274eff6f3a2979be5f7faa559" gracePeriod=30 Jul 24 18:49:04.097308 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.097270 2575 generic.go:356] "Generic (PLEG): container finished" podID="3e4dd75b-b08b-4653-9059-226877783e11" containerID="2656bd1434d412a47c58b25cc2f0d75c0762833274eff6f3a2979be5f7faa559" exitCode=0 Jul 24 18:49:04.097461 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.097351 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" event={"ID":"3e4dd75b-b08b-4653-9059-226877783e11","Type":"ContainerDied","Data":"2656bd1434d412a47c58b25cc2f0d75c0762833274eff6f3a2979be5f7faa559"} Jul 24 18:49:04.205411 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.205385 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:49:04.292275 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.292244 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4dd75b-b08b-4653-9059-226877783e11-local-certs\") pod \"3e4dd75b-b08b-4653-9059-226877783e11\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " Jul 24 18:49:04.292434 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.292292 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-kubeconfig\") pod \"3e4dd75b-b08b-4653-9059-226877783e11\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " Jul 24 18:49:04.292434 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.292336 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-istio-token\") pod \"3e4dd75b-b08b-4653-9059-226877783e11\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " Jul 24 18:49:04.292434 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.292410 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-dns-cert\") pod \"3e4dd75b-b08b-4653-9059-226877783e11\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " Jul 24 18:49:04.292569 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.292466 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pqqp\" (UniqueName: \"kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-kube-api-access-5pqqp\") pod \"3e4dd75b-b08b-4653-9059-226877783e11\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " Jul 24 18:49:04.292569 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.292496 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-ca-configmap\") pod \"3e4dd75b-b08b-4653-9059-226877783e11\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " Jul 24 18:49:04.292569 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.292521 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-cacerts\") pod \"3e4dd75b-b08b-4653-9059-226877783e11\" (UID: \"3e4dd75b-b08b-4653-9059-226877783e11\") " Jul 24 18:49:04.293187 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.293116 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3e4dd75b-b08b-4653-9059-226877783e11" (UID: "3e4dd75b-b08b-4653-9059-226877783e11"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:49:04.295042 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.294985 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3e4dd75b-b08b-4653-9059-226877783e11" (UID: "3e4dd75b-b08b-4653-9059-226877783e11"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:49:04.295144 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.295046 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-kube-api-access-5pqqp" (OuterVolumeSpecName: "kube-api-access-5pqqp") pod "3e4dd75b-b08b-4653-9059-226877783e11" (UID: "3e4dd75b-b08b-4653-9059-226877783e11"). InnerVolumeSpecName "kube-api-access-5pqqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:49:04.295144 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.295067 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e4dd75b-b08b-4653-9059-226877783e11-local-certs" (OuterVolumeSpecName: "local-certs") pod "3e4dd75b-b08b-4653-9059-226877783e11" (UID: "3e4dd75b-b08b-4653-9059-226877783e11"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:49:04.295244 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.295196 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3e4dd75b-b08b-4653-9059-226877783e11" (UID: "3e4dd75b-b08b-4653-9059-226877783e11"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:49:04.295244 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.295213 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e4dd75b-b08b-4653-9059-226877783e11" (UID: "3e4dd75b-b08b-4653-9059-226877783e11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:49:04.295412 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.295393 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-cacerts" (OuterVolumeSpecName: "cacerts") pod "3e4dd75b-b08b-4653-9059-226877783e11" (UID: "3e4dd75b-b08b-4653-9059-226877783e11"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:49:04.393990 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.393961 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e4dd75b-b08b-4653-9059-226877783e11-local-certs\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:49:04.393990 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.393987 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-kubeconfig\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:49:04.393990 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.393996 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-istio-token\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:49:04.394240 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.394004 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-dns-cert\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:49:04.394240 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.394013 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pqqp\" (UniqueName: \"kubernetes.io/projected/3e4dd75b-b08b-4653-9059-226877783e11-kube-api-access-5pqqp\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:49:04.394240 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.394022 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3e4dd75b-b08b-4653-9059-226877783e11-istio-csr-ca-configmap\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:49:04.394240 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:04.394030 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3e4dd75b-b08b-4653-9059-226877783e11-cacerts\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:49:05.103477 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:05.103447 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" Jul 24 18:49:05.103998 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:05.103473 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb" event={"ID":"3e4dd75b-b08b-4653-9059-226877783e11","Type":"ContainerDied","Data":"f40c5dc2b32a3f6918ced2e2eed6f901d99b7a389d5b90257f47e75dbf8c0c8a"} Jul 24 18:49:05.103998 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:05.103542 2575 scope.go:117] "RemoveContainer" containerID="2656bd1434d412a47c58b25cc2f0d75c0762833274eff6f3a2979be5f7faa559" Jul 24 18:49:05.105260 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:05.105231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" event={"ID":"421e836f-94c8-469e-9af3-8b8732d6282d","Type":"ContainerStarted","Data":"ce4ceed345142012f9c379061fec83997dcf39f35ce6d56726a0580b38ebfada"} Jul 24 18:49:05.105431 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:05.105307 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:05.138287 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:05.138261 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb"] Jul 24 18:49:05.152814 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:05.152791 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtzgb"] Jul 24 18:49:05.182031 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:05.181979 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" podStartSLOduration=2.650413734 podStartE2EDuration="5.181963816s" totalImagesPullingTime="2.531550082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:49:05.181386374 +0000 UTC m=+340.257898950" watchObservedRunningTime="2026-07-24 18:49:05.181963816 +0000 UTC m=+340.258476374" Jul 24 18:49:05.417125 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:05.417093 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e4dd75b-b08b-4653-9059-226877783e11" path="/var/lib/kubelet/pods/3e4dd75b-b08b-4653-9059-226877783e11/volumes" Jul 24 18:49:06.048120 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:06.048091 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-45v4h" Jul 24 18:49:06.050601 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:06.050578 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-cw2kq" Jul 24 18:49:36.116329 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:36.116292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-7gzl8" Jul 24 18:49:50.195985 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.195943 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjc6l"] Jul 24 18:49:50.196465 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.196411 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e4dd75b-b08b-4653-9059-226877783e11" containerName="discovery" Jul 24 18:49:50.196465 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.196428 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e4dd75b-b08b-4653-9059-226877783e11" containerName="discovery" Jul 24 18:49:50.196637 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.196542 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e4dd75b-b08b-4653-9059-226877783e11" containerName="discovery" Jul 24 18:49:50.199191 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.199171 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" Jul 24 18:49:50.201814 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.201790 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 18:49:50.203075 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.203055 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 18:49:50.203180 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.203092 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-gjn92\"" Jul 24 18:49:50.208332 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.208304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjc6l"] Jul 24 18:49:50.273915 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.273873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcb6x\" (UniqueName: \"kubernetes.io/projected/3b049980-ec44-47ab-af66-f5c210386d5d-kube-api-access-gcb6x\") pod \"kuadrant-operator-catalog-gjc6l\" (UID: \"3b049980-ec44-47ab-af66-f5c210386d5d\") " pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" Jul 24 18:49:50.375143 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.375107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcb6x\" (UniqueName: \"kubernetes.io/projected/3b049980-ec44-47ab-af66-f5c210386d5d-kube-api-access-gcb6x\") pod \"kuadrant-operator-catalog-gjc6l\" (UID: \"3b049980-ec44-47ab-af66-f5c210386d5d\") " pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" Jul 24 18:49:50.383889 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.383866 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcb6x\" (UniqueName: \"kubernetes.io/projected/3b049980-ec44-47ab-af66-f5c210386d5d-kube-api-access-gcb6x\") pod \"kuadrant-operator-catalog-gjc6l\" (UID: \"3b049980-ec44-47ab-af66-f5c210386d5d\") " pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" Jul 24 18:49:50.511423 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.511328 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" Jul 24 18:49:50.557507 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.557467 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjc6l"] Jul 24 18:49:50.648178 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.648154 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjc6l"] Jul 24 18:49:50.650955 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:49:50.650927 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b049980_ec44_47ab_af66_f5c210386d5d.slice/crio-9b3b760231dfba84349b78cc5b901ceb58633962dc60d49fff4dd98e435a06f2 WatchSource:0}: Error finding container 9b3b760231dfba84349b78cc5b901ceb58633962dc60d49fff4dd98e435a06f2: Status 404 returned error can't find the container with id 9b3b760231dfba84349b78cc5b901ceb58633962dc60d49fff4dd98e435a06f2 Jul 24 18:49:50.766198 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.766124 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j8c9x"] Jul 24 18:49:50.770201 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.770183 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" Jul 24 18:49:50.777646 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.777628 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j8c9x"] Jul 24 18:49:50.878912 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.878881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pklxg\" (UniqueName: \"kubernetes.io/projected/040b4576-d70e-461b-bb8b-315609527182-kube-api-access-pklxg\") pod \"kuadrant-operator-catalog-j8c9x\" (UID: \"040b4576-d70e-461b-bb8b-315609527182\") " pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" Jul 24 18:49:50.981139 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.981100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pklxg\" (UniqueName: \"kubernetes.io/projected/040b4576-d70e-461b-bb8b-315609527182-kube-api-access-pklxg\") pod \"kuadrant-operator-catalog-j8c9x\" (UID: \"040b4576-d70e-461b-bb8b-315609527182\") " pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" Jul 24 18:49:50.991127 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:50.991098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pklxg\" (UniqueName: \"kubernetes.io/projected/040b4576-d70e-461b-bb8b-315609527182-kube-api-access-pklxg\") pod \"kuadrant-operator-catalog-j8c9x\" (UID: \"040b4576-d70e-461b-bb8b-315609527182\") " pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" Jul 24 18:49:51.080530 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:51.080460 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" Jul 24 18:49:51.220438 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:51.220408 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-j8c9x"] Jul 24 18:49:51.236529 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:49:51.236498 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040b4576_d70e_461b_bb8b_315609527182.slice/crio-5939491e06e88f854367a16268ae426411e6807f59bd58e6db98669632834d21 WatchSource:0}: Error finding container 5939491e06e88f854367a16268ae426411e6807f59bd58e6db98669632834d21: Status 404 returned error can't find the container with id 5939491e06e88f854367a16268ae426411e6807f59bd58e6db98669632834d21 Jul 24 18:49:51.315603 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:51.315561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" event={"ID":"3b049980-ec44-47ab-af66-f5c210386d5d","Type":"ContainerStarted","Data":"9b3b760231dfba84349b78cc5b901ceb58633962dc60d49fff4dd98e435a06f2"} Jul 24 18:49:51.316788 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:51.316760 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" event={"ID":"040b4576-d70e-461b-bb8b-315609527182","Type":"ContainerStarted","Data":"5939491e06e88f854367a16268ae426411e6807f59bd58e6db98669632834d21"} Jul 24 18:49:53.329845 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:53.329791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" event={"ID":"3b049980-ec44-47ab-af66-f5c210386d5d","Type":"ContainerStarted","Data":"ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0"} Jul 24 18:49:53.330353 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:53.329837 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" podUID="3b049980-ec44-47ab-af66-f5c210386d5d" containerName="registry-server" containerID="cri-o://ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0" gracePeriod=2 Jul 24 18:49:53.331356 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:53.331332 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" event={"ID":"040b4576-d70e-461b-bb8b-315609527182","Type":"ContainerStarted","Data":"be75217031cf51119cee0ddc012a43a4fd41ed4fb541b3399238bd7d4f6be760"} Jul 24 18:49:53.345261 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:53.345222 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" podStartSLOduration=1.250393254 podStartE2EDuration="3.345209085s" totalImagesPullingTime="2.094815831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:49:53.344095385 +0000 UTC m=+388.420607944" watchObservedRunningTime="2026-07-24 18:49:53.345209085 +0000 UTC m=+388.421721691" Jul 24 18:49:53.359723 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:53.359686 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" podStartSLOduration=1.847295272 podStartE2EDuration="3.359672672s" totalImagesPullingTime="1.5123774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:49:53.35871352 +0000 UTC m=+388.435226091" watchObservedRunningTime="2026-07-24 18:49:53.359672672 +0000 UTC m=+388.436185230" Jul 24 18:49:53.583577 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:53.583519 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" Jul 24 18:49:53.720033 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:53.720003 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcb6x\" (UniqueName: \"kubernetes.io/projected/3b049980-ec44-47ab-af66-f5c210386d5d-kube-api-access-gcb6x\") pod \"3b049980-ec44-47ab-af66-f5c210386d5d\" (UID: \"3b049980-ec44-47ab-af66-f5c210386d5d\") " Jul 24 18:49:53.722237 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:53.722207 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b049980-ec44-47ab-af66-f5c210386d5d-kube-api-access-gcb6x" (OuterVolumeSpecName: "kube-api-access-gcb6x") pod "3b049980-ec44-47ab-af66-f5c210386d5d" (UID: "3b049980-ec44-47ab-af66-f5c210386d5d"). InnerVolumeSpecName "kube-api-access-gcb6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:49:53.820787 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:53.820729 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcb6x\" (UniqueName: \"kubernetes.io/projected/3b049980-ec44-47ab-af66-f5c210386d5d-kube-api-access-gcb6x\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:49:54.337363 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:54.337328 2575 generic.go:356] "Generic (PLEG): container finished" podID="3b049980-ec44-47ab-af66-f5c210386d5d" containerID="ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0" exitCode=0 Jul 24 18:49:54.337826 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:54.337374 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" Jul 24 18:49:54.337826 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:54.337423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" event={"ID":"3b049980-ec44-47ab-af66-f5c210386d5d","Type":"ContainerDied","Data":"ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0"} Jul 24 18:49:54.337826 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:54.337470 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gjc6l" event={"ID":"3b049980-ec44-47ab-af66-f5c210386d5d","Type":"ContainerDied","Data":"9b3b760231dfba84349b78cc5b901ceb58633962dc60d49fff4dd98e435a06f2"} Jul 24 18:49:54.337826 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:54.337491 2575 scope.go:117] "RemoveContainer" containerID="ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0" Jul 24 18:49:54.349783 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:54.349764 2575 scope.go:117] "RemoveContainer" containerID="ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0" Jul 24 18:49:54.350035 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:49:54.350016 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0\": container with ID starting with ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0 not found: ID does not exist" containerID="ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0" Jul 24 18:49:54.350083 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:54.350046 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0"} err="failed to get container status \"ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0\": rpc error: code = NotFound desc = could not find container \"ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0\": container with ID starting with ee61529d17f74df6da720af2e838f0b75e084fa5e25ef9eb094b67e6a4ee05c0 not found: ID does not exist" Jul 24 18:49:54.360015 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:54.359972 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjc6l"] Jul 24 18:49:54.367465 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:54.367446 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjc6l"] Jul 24 18:49:55.417534 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:49:55.417496 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b049980-ec44-47ab-af66-f5c210386d5d" path="/var/lib/kubelet/pods/3b049980-ec44-47ab-af66-f5c210386d5d/volumes" Jul 24 18:50:01.081091 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:01.081054 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" Jul 24 18:50:01.081616 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:01.081120 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" Jul 24 18:50:01.103465 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:01.103437 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" Jul 24 18:50:01.395702 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:01.395675 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-j8c9x" Jul 24 18:50:21.711699 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.711666 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl"] Jul 24 18:50:21.712110 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.712014 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b049980-ec44-47ab-af66-f5c210386d5d" containerName="registry-server" Jul 24 18:50:21.712110 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.712025 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b049980-ec44-47ab-af66-f5c210386d5d" containerName="registry-server" Jul 24 18:50:21.712110 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.712101 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b049980-ec44-47ab-af66-f5c210386d5d" containerName="registry-server" Jul 24 18:50:21.716345 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.716328 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl" Jul 24 18:50:21.719647 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.719623 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 18:50:21.719795 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.719673 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cpgx9\"" Jul 24 18:50:21.724409 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.724388 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl"] Jul 24 18:50:21.763197 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.763164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8jjr\" (UniqueName: \"kubernetes.io/projected/f4cddd24-220b-4224-ac3c-59328a709f56-kube-api-access-t8jjr\") pod \"dns-operator-controller-manager-864dd78c4c-nmpjl\" (UID: \"f4cddd24-220b-4224-ac3c-59328a709f56\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl" Jul 24 18:50:21.863676 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.863643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8jjr\" (UniqueName: \"kubernetes.io/projected/f4cddd24-220b-4224-ac3c-59328a709f56-kube-api-access-t8jjr\") pod \"dns-operator-controller-manager-864dd78c4c-nmpjl\" (UID: \"f4cddd24-220b-4224-ac3c-59328a709f56\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl" Jul 24 18:50:21.885798 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:21.885763 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8jjr\" (UniqueName: \"kubernetes.io/projected/f4cddd24-220b-4224-ac3c-59328a709f56-kube-api-access-t8jjr\") pod \"dns-operator-controller-manager-864dd78c4c-nmpjl\" (UID: \"f4cddd24-220b-4224-ac3c-59328a709f56\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl" Jul 24 18:50:22.027178 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:22.027099 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl" Jul 24 18:50:22.166583 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:22.166559 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl"] Jul 24 18:50:22.168647 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:50:22.168618 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cddd24_220b_4224_ac3c_59328a709f56.slice/crio-9ed872025f14991722d2f3c0a252ec7f8cc1535dc123c58e245b94c069736e43 WatchSource:0}: Error finding container 9ed872025f14991722d2f3c0a252ec7f8cc1535dc123c58e245b94c069736e43: Status 404 returned error can't find the container with id 9ed872025f14991722d2f3c0a252ec7f8cc1535dc123c58e245b94c069736e43 Jul 24 18:50:22.466644 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:22.466605 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl" event={"ID":"f4cddd24-220b-4224-ac3c-59328a709f56","Type":"ContainerStarted","Data":"9ed872025f14991722d2f3c0a252ec7f8cc1535dc123c58e245b94c069736e43"} Jul 24 18:50:23.970954 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.970920 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f6bf56ff8-gnvh4"] Jul 24 18:50:23.975321 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.975291 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:23.979312 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.979282 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 18:50:23.979554 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.979536 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 18:50:23.979725 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.979703 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rw2c6\"" Jul 24 18:50:23.980665 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.980601 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 18:50:23.980665 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.980633 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 18:50:23.980853 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.980640 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 18:50:23.980956 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.980917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-console-config\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:23.981039 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.980996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-oauth-serving-cert\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:23.981104 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.981066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89d938c4-ee89-45f8-b063-0e421b7b64e4-console-oauth-config\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:23.981163 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.981116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89d938c4-ee89-45f8-b063-0e421b7b64e4-console-serving-cert\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:23.981215 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.981182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-service-ca\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:23.981267 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.981227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-trusted-ca-bundle\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:23.981323 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.981269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wp4b\" (UniqueName: \"kubernetes.io/projected/89d938c4-ee89-45f8-b063-0e421b7b64e4-kube-api-access-9wp4b\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:23.996322 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.996291 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 18:50:23.997548 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:23.997525 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6bf56ff8-gnvh4"] Jul 24 18:50:24.082411 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.082373 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-console-config\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.082618 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.082432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-oauth-serving-cert\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.082618 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.082468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89d938c4-ee89-45f8-b063-0e421b7b64e4-console-oauth-config\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.082618 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.082504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89d938c4-ee89-45f8-b063-0e421b7b64e4-console-serving-cert\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.082618 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.082542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-service-ca\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.082618 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.082576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-trusted-ca-bundle\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.082618 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.082613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wp4b\" (UniqueName: \"kubernetes.io/projected/89d938c4-ee89-45f8-b063-0e421b7b64e4-kube-api-access-9wp4b\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.083305 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.083270 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-console-config\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.083432 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.083374 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-service-ca\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.083575 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.083548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-oauth-serving-cert\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.083769 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.083552 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89d938c4-ee89-45f8-b063-0e421b7b64e4-trusted-ca-bundle\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.085699 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.085653 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89d938c4-ee89-45f8-b063-0e421b7b64e4-console-oauth-config\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.085878 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.085851 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89d938c4-ee89-45f8-b063-0e421b7b64e4-console-serving-cert\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.098655 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.098620 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wp4b\" (UniqueName: \"kubernetes.io/projected/89d938c4-ee89-45f8-b063-0e421b7b64e4-kube-api-access-9wp4b\") pod \"console-5f6bf56ff8-gnvh4\" (UID: \"89d938c4-ee89-45f8-b063-0e421b7b64e4\") " pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.296867 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.296832 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:24.428458 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.428433 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6bf56ff8-gnvh4"] Jul 24 18:50:24.431201 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:50:24.431176 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d938c4_ee89_45f8_b063_0e421b7b64e4.slice/crio-398f004ba5f95a735ed7f028f61fba9cc2887b6f25d94c5c97c2a2c1aeefd39c WatchSource:0}: Error finding container 398f004ba5f95a735ed7f028f61fba9cc2887b6f25d94c5c97c2a2c1aeefd39c: Status 404 returned error can't find the container with id 398f004ba5f95a735ed7f028f61fba9cc2887b6f25d94c5c97c2a2c1aeefd39c Jul 24 18:50:24.478536 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.478503 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl" event={"ID":"f4cddd24-220b-4224-ac3c-59328a709f56","Type":"ContainerStarted","Data":"5a905a915b8fc94e4185f433fcf439b179faf43c39c7f4ea8179c233df62f334"} Jul 24 18:50:24.478659 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.478566 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl" Jul 24 18:50:24.479873 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.479841 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6bf56ff8-gnvh4" event={"ID":"89d938c4-ee89-45f8-b063-0e421b7b64e4","Type":"ContainerStarted","Data":"398f004ba5f95a735ed7f028f61fba9cc2887b6f25d94c5c97c2a2c1aeefd39c"} Jul 24 18:50:24.496246 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:24.496198 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl" podStartSLOduration=1.47858073 podStartE2EDuration="3.49617918s" totalImagesPullingTime="2.01759845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:50:24.494753399 +0000 UTC m=+419.571265976" watchObservedRunningTime="2026-07-24 18:50:24.49617918 +0000 UTC m=+419.572691738" Jul 24 18:50:25.485516 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:25.485476 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6bf56ff8-gnvh4" event={"ID":"89d938c4-ee89-45f8-b063-0e421b7b64e4","Type":"ContainerStarted","Data":"f20e7571554c2b3f8a046344b30bf274d6564b0299b25bf1f987d4bebe9d6de6"} Jul 24 18:50:25.505049 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:25.505008 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f6bf56ff8-gnvh4" podStartSLOduration=2.504993288 podStartE2EDuration="2.504993288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:50:25.503194681 +0000 UTC m=+420.579707239" watchObservedRunningTime="2026-07-24 18:50:25.504993288 +0000 UTC m=+420.581505845" Jul 24 18:50:27.178952 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.178914 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2"] Jul 24 18:50:27.186755 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.186708 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:27.190936 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.190907 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f7xtj\"" Jul 24 18:50:27.207807 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.207772 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2"] Jul 24 18:50:27.211238 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.211211 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/243d2189-7a5d-41f3-a72e-941351994504-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" (UID: \"243d2189-7a5d-41f3-a72e-941351994504\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:27.211341 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.211259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpb4b\" (UniqueName: \"kubernetes.io/projected/243d2189-7a5d-41f3-a72e-941351994504-kube-api-access-hpb4b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" (UID: \"243d2189-7a5d-41f3-a72e-941351994504\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:27.312507 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.312464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/243d2189-7a5d-41f3-a72e-941351994504-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" (UID: \"243d2189-7a5d-41f3-a72e-941351994504\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:27.312691 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.312535 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpb4b\" (UniqueName: \"kubernetes.io/projected/243d2189-7a5d-41f3-a72e-941351994504-kube-api-access-hpb4b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" (UID: \"243d2189-7a5d-41f3-a72e-941351994504\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:27.312857 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.312838 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/243d2189-7a5d-41f3-a72e-941351994504-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" (UID: \"243d2189-7a5d-41f3-a72e-941351994504\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:27.325592 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.325560 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpb4b\" (UniqueName: \"kubernetes.io/projected/243d2189-7a5d-41f3-a72e-941351994504-kube-api-access-hpb4b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" (UID: \"243d2189-7a5d-41f3-a72e-941351994504\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:27.498007 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.497917 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:27.646371 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:27.646261 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2"] Jul 24 18:50:27.648898 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:50:27.648870 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243d2189_7a5d_41f3_a72e_941351994504.slice/crio-332173fc388d84b84b587304b303c4641144e0c3796380f4f8faf19885a306ce WatchSource:0}: Error finding container 332173fc388d84b84b587304b303c4641144e0c3796380f4f8faf19885a306ce: Status 404 returned error can't find the container with id 332173fc388d84b84b587304b303c4641144e0c3796380f4f8faf19885a306ce Jul 24 18:50:28.502074 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:28.502034 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" event={"ID":"243d2189-7a5d-41f3-a72e-941351994504","Type":"ContainerStarted","Data":"332173fc388d84b84b587304b303c4641144e0c3796380f4f8faf19885a306ce"} Jul 24 18:50:30.477752 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:30.477717 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c"] Jul 24 18:50:30.481413 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:30.481383 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c" Jul 24 18:50:30.484399 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:30.484374 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n8hrq\"" Jul 24 18:50:30.499653 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:30.499624 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c"] Jul 24 18:50:30.540125 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:30.540091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtdwr\" (UniqueName: \"kubernetes.io/projected/dbf1c171-593d-4237-8c93-f068ee65eb8b-kube-api-access-qtdwr\") pod \"authorino-operator-75cc8d5cb7-c5q5c\" (UID: \"dbf1c171-593d-4237-8c93-f068ee65eb8b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c" Jul 24 18:50:30.641339 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:30.641305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdwr\" (UniqueName: \"kubernetes.io/projected/dbf1c171-593d-4237-8c93-f068ee65eb8b-kube-api-access-qtdwr\") pod \"authorino-operator-75cc8d5cb7-c5q5c\" (UID: \"dbf1c171-593d-4237-8c93-f068ee65eb8b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c" Jul 24 18:50:30.656126 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:30.656095 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtdwr\" (UniqueName: \"kubernetes.io/projected/dbf1c171-593d-4237-8c93-f068ee65eb8b-kube-api-access-qtdwr\") pod \"authorino-operator-75cc8d5cb7-c5q5c\" (UID: \"dbf1c171-593d-4237-8c93-f068ee65eb8b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c" Jul 24 18:50:30.794310 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:30.794219 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c" Jul 24 18:50:30.953000 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:30.952975 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c"] Jul 24 18:50:30.955424 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:50:30.955399 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf1c171_593d_4237_8c93_f068ee65eb8b.slice/crio-fc687ce4d80cfb2151ffc391f7df97c1418dd21b21b0c8daf0112d332c0bd111 WatchSource:0}: Error finding container fc687ce4d80cfb2151ffc391f7df97c1418dd21b21b0c8daf0112d332c0bd111: Status 404 returned error can't find the container with id fc687ce4d80cfb2151ffc391f7df97c1418dd21b21b0c8daf0112d332c0bd111 Jul 24 18:50:31.519826 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:31.519789 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c" event={"ID":"dbf1c171-593d-4237-8c93-f068ee65eb8b","Type":"ContainerStarted","Data":"fc687ce4d80cfb2151ffc391f7df97c1418dd21b21b0c8daf0112d332c0bd111"} Jul 24 18:50:32.531967 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:32.531930 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" event={"ID":"243d2189-7a5d-41f3-a72e-941351994504","Type":"ContainerStarted","Data":"b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816"} Jul 24 18:50:32.532393 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:32.532063 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:32.552846 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:32.552763 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" podStartSLOduration=0.796072841 podStartE2EDuration="5.552728839s" totalImagesPullingTime="4.756655998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:50:32.551543293 +0000 UTC m=+427.628055853" watchObservedRunningTime="2026-07-24 18:50:32.552728839 +0000 UTC m=+427.629241398" Jul 24 18:50:33.429116 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:33.429086 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86"] Jul 24 18:50:33.436084 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:33.436056 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:33.438852 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:33.438829 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-gqskq\"" Jul 24 18:50:33.443632 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:33.443603 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86"] Jul 24 18:50:33.569963 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:33.569925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prpbd\" (UniqueName: \"kubernetes.io/projected/a53e8245-0d1c-4538-bcb5-dd3826b862b6-kube-api-access-prpbd\") pod \"limitador-operator-controller-manager-859d7b78bd-kpz86\" (UID: \"a53e8245-0d1c-4538-bcb5-dd3826b862b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:33.671352 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:33.671314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpbd\" (UniqueName: \"kubernetes.io/projected/a53e8245-0d1c-4538-bcb5-dd3826b862b6-kube-api-access-prpbd\") pod \"limitador-operator-controller-manager-859d7b78bd-kpz86\" (UID: \"a53e8245-0d1c-4538-bcb5-dd3826b862b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:33.680821 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:33.680729 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpbd\" (UniqueName: \"kubernetes.io/projected/a53e8245-0d1c-4538-bcb5-dd3826b862b6-kube-api-access-prpbd\") pod \"limitador-operator-controller-manager-859d7b78bd-kpz86\" (UID: \"a53e8245-0d1c-4538-bcb5-dd3826b862b6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:33.748753 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:33.748701 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:34.296971 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:34.296943 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:34.297199 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:34.297119 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:34.302644 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:34.302623 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:34.309276 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:34.309252 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86"] Jul 24 18:50:34.311114 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:50:34.311083 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53e8245_0d1c_4538_bcb5_dd3826b862b6.slice/crio-9e59158d2e74b1854a8505c5a1f2b445dccaed1e4aa4f88087ab8e1c16e24ae3 WatchSource:0}: Error finding container 9e59158d2e74b1854a8505c5a1f2b445dccaed1e4aa4f88087ab8e1c16e24ae3: Status 404 returned error can't find the container with id 9e59158d2e74b1854a8505c5a1f2b445dccaed1e4aa4f88087ab8e1c16e24ae3 Jul 24 18:50:34.543773 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:34.543667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" event={"ID":"a53e8245-0d1c-4538-bcb5-dd3826b862b6","Type":"ContainerStarted","Data":"9e59158d2e74b1854a8505c5a1f2b445dccaed1e4aa4f88087ab8e1c16e24ae3"} Jul 24 18:50:34.545250 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:34.545211 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c" event={"ID":"dbf1c171-593d-4237-8c93-f068ee65eb8b","Type":"ContainerStarted","Data":"9142edc12a05ff4d6ed69b1ff367081afa19dbeb1a10c261385cef9774ae9057"} Jul 24 18:50:34.549121 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:34.549096 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f6bf56ff8-gnvh4" Jul 24 18:50:34.564815 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:34.564770 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c" podStartSLOduration=1.31806977 podStartE2EDuration="4.564728066s" totalImagesPullingTime="3.246658296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:50:34.563006718 +0000 UTC m=+429.639519277" watchObservedRunningTime="2026-07-24 18:50:34.564728066 +0000 UTC m=+429.641240624" Jul 24 18:50:35.488297 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:35.488265 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-nmpjl" Jul 24 18:50:35.549973 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:35.549940 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c" Jul 24 18:50:36.556315 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:36.556270 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" event={"ID":"a53e8245-0d1c-4538-bcb5-dd3826b862b6","Type":"ContainerStarted","Data":"4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3"} Jul 24 18:50:36.556793 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:36.556675 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:36.577167 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:36.577119 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" podStartSLOduration=1.8866964720000001 podStartE2EDuration="3.57710457s" totalImagesPullingTime="1.690408098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:50:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:50:36.575552734 +0000 UTC m=+431.652065295" watchObservedRunningTime="2026-07-24 18:50:36.57710457 +0000 UTC m=+431.653617196" Jul 24 18:50:43.539244 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:43.539214 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:45.191602 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.191557 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2"] Jul 24 18:50:45.192018 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.191799 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" podUID="243d2189-7a5d-41f3-a72e-941351994504" containerName="manager" containerID="cri-o://b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816" gracePeriod=2 Jul 24 18:50:45.206636 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.206611 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2"] Jul 24 18:50:45.220564 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.220524 2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"243d2189-7a5d-41f3-a72e-941351994504\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-njgj2\": pods \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:45.231820 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.231791 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86"] Jul 24 18:50:45.232035 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.232008 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" containerName="manager" containerID="cri-o://4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3" gracePeriod=2 Jul 24 18:50:45.234005 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.233984 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:45.234937 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.234905 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="243d2189-7a5d-41f3-a72e-941351994504" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:45.243913 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.243891 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr"] Jul 24 18:50:45.244284 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.244271 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="243d2189-7a5d-41f3-a72e-941351994504" containerName="manager" Jul 24 18:50:45.244377 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.244286 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="243d2189-7a5d-41f3-a72e-941351994504" containerName="manager" Jul 24 18:50:45.244422 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.244377 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="243d2189-7a5d-41f3-a72e-941351994504" containerName="manager" Jul 24 18:50:45.247838 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.247818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:45.256118 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.256094 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86"] Jul 24 18:50:45.268246 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.268192 2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"a53e8245-0d1c-4538-bcb5-dd3826b862b6\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-kpz86\": pods \"limitador-operator-controller-manager-859d7b78bd-kpz86\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:45.270178 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.270157 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr"] Jul 24 18:50:45.279230 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.279208 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk"] Jul 24 18:50:45.279636 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.279621 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" containerName="manager" Jul 24 18:50:45.279701 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.279638 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" containerName="manager" Jul 24 18:50:45.279772 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.279722 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" containerName="manager" Jul 24 18:50:45.282533 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.282517 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk" Jul 24 18:50:45.291545 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.291506 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="243d2189-7a5d-41f3-a72e-941351994504" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:45.294191 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.294169 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk"] Jul 24 18:50:45.294274 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.294248 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpz86\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:45.296714 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.296688 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="243d2189-7a5d-41f3-a72e-941351994504" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:45.299314 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.299288 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpz86\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:45.377188 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.377154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-29lbr\" (UID: \"f90f82a5-4b3c-463d-8c4b-3d681ab5515c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:45.377341 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.377275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnzhx\" (UniqueName: \"kubernetes.io/projected/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-kube-api-access-pnzhx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-29lbr\" (UID: \"f90f82a5-4b3c-463d-8c4b-3d681ab5515c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:45.377341 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.377314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6zk\" (UniqueName: \"kubernetes.io/projected/42836617-36e5-4fec-8cf8-5e50a3dd749b-kube-api-access-df6zk\") pod \"limitador-operator-controller-manager-859d7b78bd-tlzpk\" (UID: \"42836617-36e5-4fec-8cf8-5e50a3dd749b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk" Jul 24 18:50:45.416405 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.416361 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-njgj2\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="243d2189-7a5d-41f3-a72e-941351994504" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:45.420075 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.420040 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpz86\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:45.477798 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.477768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnzhx\" (UniqueName: \"kubernetes.io/projected/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-kube-api-access-pnzhx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-29lbr\" (UID: \"f90f82a5-4b3c-463d-8c4b-3d681ab5515c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:45.477915 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.477807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6zk\" (UniqueName: \"kubernetes.io/projected/42836617-36e5-4fec-8cf8-5e50a3dd749b-kube-api-access-df6zk\") pod \"limitador-operator-controller-manager-859d7b78bd-tlzpk\" (UID: \"42836617-36e5-4fec-8cf8-5e50a3dd749b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk" Jul 24 18:50:45.477915 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.477855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-29lbr\" (UID: \"f90f82a5-4b3c-463d-8c4b-3d681ab5515c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:45.478198 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.478181 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-29lbr\" (UID: \"f90f82a5-4b3c-463d-8c4b-3d681ab5515c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:45.484651 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.484633 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:45.487473 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.487454 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:45.489363 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.489340 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6zk\" (UniqueName: \"kubernetes.io/projected/42836617-36e5-4fec-8cf8-5e50a3dd749b-kube-api-access-df6zk\") pod \"limitador-operator-controller-manager-859d7b78bd-tlzpk\" (UID: \"42836617-36e5-4fec-8cf8-5e50a3dd749b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk" Jul 24 18:50:45.489442 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.489425 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnzhx\" (UniqueName: \"kubernetes.io/projected/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-kube-api-access-pnzhx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-29lbr\" (UID: \"f90f82a5-4b3c-463d-8c4b-3d681ab5515c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:45.578731 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.578699 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpb4b\" (UniqueName: \"kubernetes.io/projected/243d2189-7a5d-41f3-a72e-941351994504-kube-api-access-hpb4b\") pod \"243d2189-7a5d-41f3-a72e-941351994504\" (UID: \"243d2189-7a5d-41f3-a72e-941351994504\") " Jul 24 18:50:45.578731 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.578732 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/243d2189-7a5d-41f3-a72e-941351994504-extensions-socket-volume\") pod \"243d2189-7a5d-41f3-a72e-941351994504\" (UID: \"243d2189-7a5d-41f3-a72e-941351994504\") " Jul 24 18:50:45.578940 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.578793 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prpbd\" (UniqueName: \"kubernetes.io/projected/a53e8245-0d1c-4538-bcb5-dd3826b862b6-kube-api-access-prpbd\") pod \"a53e8245-0d1c-4538-bcb5-dd3826b862b6\" (UID: \"a53e8245-0d1c-4538-bcb5-dd3826b862b6\") " Jul 24 18:50:45.579220 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.579190 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/243d2189-7a5d-41f3-a72e-941351994504-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "243d2189-7a5d-41f3-a72e-941351994504" (UID: "243d2189-7a5d-41f3-a72e-941351994504"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:50:45.580895 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.580874 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243d2189-7a5d-41f3-a72e-941351994504-kube-api-access-hpb4b" (OuterVolumeSpecName: "kube-api-access-hpb4b") pod "243d2189-7a5d-41f3-a72e-941351994504" (UID: "243d2189-7a5d-41f3-a72e-941351994504"). InnerVolumeSpecName "kube-api-access-hpb4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:50:45.580895 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.580879 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53e8245-0d1c-4538-bcb5-dd3826b862b6-kube-api-access-prpbd" (OuterVolumeSpecName: "kube-api-access-prpbd") pod "a53e8245-0d1c-4538-bcb5-dd3826b862b6" (UID: "a53e8245-0d1c-4538-bcb5-dd3826b862b6"). InnerVolumeSpecName "kube-api-access-prpbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:50:45.598989 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.598960 2575 generic.go:356] "Generic (PLEG): container finished" podID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" containerID="4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3" exitCode=0 Jul 24 18:50:45.599094 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.599005 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:45.599094 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.599053 2575 scope.go:117] "RemoveContainer" containerID="4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3" Jul 24 18:50:45.600419 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.600400 2575 generic.go:356] "Generic (PLEG): container finished" podID="243d2189-7a5d-41f3-a72e-941351994504" containerID="b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816" exitCode=0 Jul 24 18:50:45.600679 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.600442 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-njgj2" Jul 24 18:50:45.607853 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.607826 2575 scope.go:117] "RemoveContainer" containerID="4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3" Jul 24 18:50:45.608117 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.608092 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3\": container with ID starting with 4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3 not found: ID does not exist" containerID="4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3" Jul 24 18:50:45.608200 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.608129 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3"} err="failed to get container status \"4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3\": rpc error: code = NotFound desc = could not find container \"4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3\": container with ID starting with 4df1db25c0799474c6ba69510407332347663fc9af4a3ab1bba1a52cc36daaa3 not found: ID does not exist" Jul 24 18:50:45.608200 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.608154 2575 scope.go:117] "RemoveContainer" containerID="b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816" Jul 24 18:50:45.614486 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.614463 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpz86\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:45.616308 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.616294 2575 scope.go:117] "RemoveContainer" containerID="b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816" Jul 24 18:50:45.616550 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.616533 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816\": container with ID starting with b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816 not found: ID does not exist" containerID="b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816" Jul 24 18:50:45.616596 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.616559 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816"} err="failed to get container status \"b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816\": rpc error: code = NotFound desc = could not find container \"b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816\": container with ID starting with b0a9fee827a093c230cfa12660da7c39cb60b660014deff69b618a760d930816 not found: ID does not exist" Jul 24 18:50:45.616596 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:45.616563 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpz86\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:45.631300 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.631281 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:45.637269 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.637249 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk" Jul 24 18:50:45.680137 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.680068 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prpbd\" (UniqueName: \"kubernetes.io/projected/a53e8245-0d1c-4538-bcb5-dd3826b862b6-kube-api-access-prpbd\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:50:45.680137 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.680096 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpb4b\" (UniqueName: \"kubernetes.io/projected/243d2189-7a5d-41f3-a72e-941351994504-kube-api-access-hpb4b\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:50:45.680137 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.680106 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/243d2189-7a5d-41f3-a72e-941351994504-extensions-socket-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:50:45.792772 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.792723 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr"] Jul 24 18:50:45.795567 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:50:45.795537 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90f82a5_4b3c_463d_8c4b_3d681ab5515c.slice/crio-7404dcfcc6564b4b9d41728288135ce407fc4ae3846da3e24f687b16861a2cf7 WatchSource:0}: Error finding container 7404dcfcc6564b4b9d41728288135ce407fc4ae3846da3e24f687b16861a2cf7: Status 404 returned error can't find the container with id 7404dcfcc6564b4b9d41728288135ce407fc4ae3846da3e24f687b16861a2cf7 Jul 24 18:50:45.805765 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:45.805729 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk"] Jul 24 18:50:45.807277 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:50:45.807250 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42836617_36e5_4fec_8cf8_5e50a3dd749b.slice/crio-062c19b758e79897bc76ab3271711ae8b595db38e43e5a12891e79fb0ef4a4f0 WatchSource:0}: Error finding container 062c19b758e79897bc76ab3271711ae8b595db38e43e5a12891e79fb0ef4a4f0: Status 404 returned error can't find the container with id 062c19b758e79897bc76ab3271711ae8b595db38e43e5a12891e79fb0ef4a4f0 Jul 24 18:50:46.158467 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.158440 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4"] Jul 24 18:50:46.162005 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.161985 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 18:50:46.164525 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:46.164494 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpz86\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:46.176691 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.176670 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4"] Jul 24 18:50:46.285718 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.285674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9hj\" (UniqueName: \"kubernetes.io/projected/4b2cc625-0c89-4443-8346-7c6362cbbcb3-kube-api-access-gw9hj\") pod \"kuadrant-operator-controller-manager-86878c769f-l42t4\" (UID: \"4b2cc625-0c89-4443-8346-7c6362cbbcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 18:50:46.286099 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.285784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b2cc625-0c89-4443-8346-7c6362cbbcb3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l42t4\" (UID: \"4b2cc625-0c89-4443-8346-7c6362cbbcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 18:50:46.386921 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.386834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b2cc625-0c89-4443-8346-7c6362cbbcb3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l42t4\" (UID: \"4b2cc625-0c89-4443-8346-7c6362cbbcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 18:50:46.387075 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.386938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9hj\" (UniqueName: \"kubernetes.io/projected/4b2cc625-0c89-4443-8346-7c6362cbbcb3-kube-api-access-gw9hj\") pod \"kuadrant-operator-controller-manager-86878c769f-l42t4\" (UID: \"4b2cc625-0c89-4443-8346-7c6362cbbcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 18:50:46.387306 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.387282 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b2cc625-0c89-4443-8346-7c6362cbbcb3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l42t4\" (UID: \"4b2cc625-0c89-4443-8346-7c6362cbbcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 18:50:46.395982 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.395957 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9hj\" (UniqueName: \"kubernetes.io/projected/4b2cc625-0c89-4443-8346-7c6362cbbcb3-kube-api-access-gw9hj\") pod \"kuadrant-operator-controller-manager-86878c769f-l42t4\" (UID: \"4b2cc625-0c89-4443-8346-7c6362cbbcb3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 18:50:46.472206 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.472141 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 18:50:46.558445 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.558415 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-c5q5c" Jul 24 18:50:46.561416 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:46.561377 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpz86\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:46.608836 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.608794 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk" event={"ID":"42836617-36e5-4fec-8cf8-5e50a3dd749b","Type":"ContainerStarted","Data":"d1c4cc6cb222990039b3f862046386849aaaf7f72538245a6600be0c6e4455cd"} Jul 24 18:50:46.608984 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.608876 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk" event={"ID":"42836617-36e5-4fec-8cf8-5e50a3dd749b","Type":"ContainerStarted","Data":"062c19b758e79897bc76ab3271711ae8b595db38e43e5a12891e79fb0ef4a4f0"} Jul 24 18:50:46.609446 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.609391 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk" Jul 24 18:50:46.612119 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:46.612087 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpz86\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:46.612606 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.612585 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" event={"ID":"f90f82a5-4b3c-463d-8c4b-3d681ab5515c","Type":"ContainerStarted","Data":"db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1"} Jul 24 18:50:46.612684 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.612622 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" event={"ID":"f90f82a5-4b3c-463d-8c4b-3d681ab5515c","Type":"ContainerStarted","Data":"7404dcfcc6564b4b9d41728288135ce407fc4ae3846da3e24f687b16861a2cf7"} Jul 24 18:50:46.612810 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.612792 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:46.628927 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.628895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4"] Jul 24 18:50:46.630626 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.630577 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk" podStartSLOduration=1.6305596869999999 podStartE2EDuration="1.630559687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:50:46.629693509 +0000 UTC m=+441.706206067" watchObservedRunningTime="2026-07-24 18:50:46.630559687 +0000 UTC m=+441.707072246" Jul 24 18:50:46.656368 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:46.656330 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-kpz86\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kpz86" Jul 24 18:50:46.657573 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:46.657528 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" podStartSLOduration=1.657512675 podStartE2EDuration="1.657512675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:50:46.654221017 +0000 UTC m=+441.730733576" watchObservedRunningTime="2026-07-24 18:50:46.657512675 +0000 UTC m=+441.734025233" Jul 24 18:50:47.418413 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:47.418382 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="243d2189-7a5d-41f3-a72e-941351994504" path="/var/lib/kubelet/pods/243d2189-7a5d-41f3-a72e-941351994504/volumes" Jul 24 18:50:47.418790 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:47.418689 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a53e8245-0d1c-4538-bcb5-dd3826b862b6" path="/var/lib/kubelet/pods/a53e8245-0d1c-4538-bcb5-dd3826b862b6/volumes" Jul 24 18:50:47.620539 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:47.620496 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" event={"ID":"4b2cc625-0c89-4443-8346-7c6362cbbcb3","Type":"ContainerStarted","Data":"f40b0d632a58f3be69563642a651b40f1813f361c0e5cce8fc8362204a623ab5"} Jul 24 18:50:47.620705 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:47.620567 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 18:50:47.620705 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:47.620589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" event={"ID":"4b2cc625-0c89-4443-8346-7c6362cbbcb3","Type":"ContainerStarted","Data":"a226c5ddb868370ce498621446de51c1a8a94dfb3a8ad2c4823e4038767ae043"} Jul 24 18:50:47.645889 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:47.645835 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" podStartSLOduration=1.6458217240000002 podStartE2EDuration="1.645821724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:50:47.643094631 +0000 UTC m=+442.719607190" watchObservedRunningTime="2026-07-24 18:50:47.645821724 +0000 UTC m=+442.722334281" Jul 24 18:50:57.622619 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:57.622589 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:57.623082 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:57.622655 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tlzpk" Jul 24 18:50:58.629433 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:58.629404 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 18:50:58.685344 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:58.685313 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr"] Jul 24 18:50:58.685512 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:58.685490 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" podUID="f90f82a5-4b3c-463d-8c4b-3d681ab5515c" containerName="manager" containerID="cri-o://db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1" gracePeriod=10 Jul 24 18:50:58.947527 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:58.947495 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:59.101060 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.101029 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-extensions-socket-volume\") pod \"f90f82a5-4b3c-463d-8c4b-3d681ab5515c\" (UID: \"f90f82a5-4b3c-463d-8c4b-3d681ab5515c\") " Jul 24 18:50:59.101236 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.101096 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnzhx\" (UniqueName: \"kubernetes.io/projected/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-kube-api-access-pnzhx\") pod \"f90f82a5-4b3c-463d-8c4b-3d681ab5515c\" (UID: \"f90f82a5-4b3c-463d-8c4b-3d681ab5515c\") " Jul 24 18:50:59.101449 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.101419 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f90f82a5-4b3c-463d-8c4b-3d681ab5515c" (UID: "f90f82a5-4b3c-463d-8c4b-3d681ab5515c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:50:59.103345 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.103322 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-kube-api-access-pnzhx" (OuterVolumeSpecName: "kube-api-access-pnzhx") pod "f90f82a5-4b3c-463d-8c4b-3d681ab5515c" (UID: "f90f82a5-4b3c-463d-8c4b-3d681ab5515c"). InnerVolumeSpecName "kube-api-access-pnzhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:50:59.202314 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.202229 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-extensions-socket-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:50:59.202314 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.202258 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnzhx\" (UniqueName: \"kubernetes.io/projected/f90f82a5-4b3c-463d-8c4b-3d681ab5515c-kube-api-access-pnzhx\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:50:59.684798 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.684765 2575 generic.go:356] "Generic (PLEG): container finished" podID="f90f82a5-4b3c-463d-8c4b-3d681ab5515c" containerID="db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1" exitCode=0 Jul 24 18:50:59.685261 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.684800 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" Jul 24 18:50:59.685261 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.684845 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" event={"ID":"f90f82a5-4b3c-463d-8c4b-3d681ab5515c","Type":"ContainerDied","Data":"db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1"} Jul 24 18:50:59.685261 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.684904 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr" event={"ID":"f90f82a5-4b3c-463d-8c4b-3d681ab5515c","Type":"ContainerDied","Data":"7404dcfcc6564b4b9d41728288135ce407fc4ae3846da3e24f687b16861a2cf7"} Jul 24 18:50:59.685261 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.684941 2575 scope.go:117] "RemoveContainer" containerID="db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1" Jul 24 18:50:59.694027 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.693864 2575 scope.go:117] "RemoveContainer" containerID="db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1" Jul 24 18:50:59.694129 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:50:59.694112 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1\": container with ID starting with db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1 not found: ID does not exist" containerID="db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1" Jul 24 18:50:59.694166 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.694138 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1"} err="failed to get container status \"db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1\": rpc error: code = NotFound desc = could not find container \"db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1\": container with ID starting with db619238db4bf218409657dbe4067e79b5772dfb5b46f055a20a68d5083096e1 not found: ID does not exist" Jul 24 18:50:59.706134 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.706111 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr"] Jul 24 18:50:59.709305 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:50:59.709284 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-29lbr"] Jul 24 18:51:01.417009 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:01.416978 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90f82a5-4b3c-463d-8c4b-3d681ab5515c" path="/var/lib/kubelet/pods/f90f82a5-4b3c-463d-8c4b-3d681ab5515c/volumes" Jul 24 18:51:17.381575 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.381542 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6rwkq"] Jul 24 18:51:17.382005 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.381940 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90f82a5-4b3c-463d-8c4b-3d681ab5515c" containerName="manager" Jul 24 18:51:17.382005 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.381951 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90f82a5-4b3c-463d-8c4b-3d681ab5515c" containerName="manager" Jul 24 18:51:17.382077 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.382027 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90f82a5-4b3c-463d-8c4b-3d681ab5515c" containerName="manager" Jul 24 18:51:17.385341 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.385323 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:17.387965 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.387937 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 18:51:17.388081 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.387983 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jwg7r\"" Jul 24 18:51:17.395298 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.395274 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6rwkq"] Jul 24 18:51:17.447251 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.447217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnvg\" (UniqueName: \"kubernetes.io/projected/09742c1d-7e12-40c1-95bf-f889a52be85e-kube-api-access-ddnvg\") pod \"limitador-limitador-64c94fd64f-6rwkq\" (UID: \"09742c1d-7e12-40c1-95bf-f889a52be85e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:17.447427 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.447261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09742c1d-7e12-40c1-95bf-f889a52be85e-config-file\") pod \"limitador-limitador-64c94fd64f-6rwkq\" (UID: \"09742c1d-7e12-40c1-95bf-f889a52be85e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:17.474900 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.474867 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6rwkq"] Jul 24 18:51:17.548728 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.548695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnvg\" (UniqueName: \"kubernetes.io/projected/09742c1d-7e12-40c1-95bf-f889a52be85e-kube-api-access-ddnvg\") pod \"limitador-limitador-64c94fd64f-6rwkq\" (UID: \"09742c1d-7e12-40c1-95bf-f889a52be85e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:17.548728 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.548761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09742c1d-7e12-40c1-95bf-f889a52be85e-config-file\") pod \"limitador-limitador-64c94fd64f-6rwkq\" (UID: \"09742c1d-7e12-40c1-95bf-f889a52be85e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:17.549355 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.549336 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09742c1d-7e12-40c1-95bf-f889a52be85e-config-file\") pod \"limitador-limitador-64c94fd64f-6rwkq\" (UID: \"09742c1d-7e12-40c1-95bf-f889a52be85e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:17.557036 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.557013 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnvg\" (UniqueName: \"kubernetes.io/projected/09742c1d-7e12-40c1-95bf-f889a52be85e-kube-api-access-ddnvg\") pod \"limitador-limitador-64c94fd64f-6rwkq\" (UID: \"09742c1d-7e12-40c1-95bf-f889a52be85e\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:17.696816 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.696779 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:17.837457 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:17.837427 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6rwkq"] Jul 24 18:51:17.840205 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:51:17.840178 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09742c1d_7e12_40c1_95bf_f889a52be85e.slice/crio-aa9c0f7ab1cb0d6e5c35206d25d6b5035679ffb830f78910fa7b033321f3b043 WatchSource:0}: Error finding container aa9c0f7ab1cb0d6e5c35206d25d6b5035679ffb830f78910fa7b033321f3b043: Status 404 returned error can't find the container with id aa9c0f7ab1cb0d6e5c35206d25d6b5035679ffb830f78910fa7b033321f3b043 Jul 24 18:51:18.780354 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:18.780317 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" event={"ID":"09742c1d-7e12-40c1-95bf-f889a52be85e","Type":"ContainerStarted","Data":"aa9c0f7ab1cb0d6e5c35206d25d6b5035679ffb830f78910fa7b033321f3b043"} Jul 24 18:51:21.806475 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:21.806437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" event={"ID":"09742c1d-7e12-40c1-95bf-f889a52be85e","Type":"ContainerStarted","Data":"d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058"} Jul 24 18:51:21.806994 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:21.806633 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:21.825090 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:21.825041 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" podStartSLOduration=1.802067148 podStartE2EDuration="4.825026999s" totalImagesPullingTime="3.022959851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:51:21.823031052 +0000 UTC m=+476.899543634" watchObservedRunningTime="2026-07-24 18:51:21.825026999 +0000 UTC m=+476.901539570" Jul 24 18:51:32.115038 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.115002 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6rwkq"] Jul 24 18:51:32.115498 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.115234 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" podUID="09742c1d-7e12-40c1-95bf-f889a52be85e" containerName="limitador" containerID="cri-o://d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058" gracePeriod=30 Jul 24 18:51:32.115926 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.115909 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:32.659324 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.659292 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:32.792554 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.792474 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddnvg\" (UniqueName: \"kubernetes.io/projected/09742c1d-7e12-40c1-95bf-f889a52be85e-kube-api-access-ddnvg\") pod \"09742c1d-7e12-40c1-95bf-f889a52be85e\" (UID: \"09742c1d-7e12-40c1-95bf-f889a52be85e\") " Jul 24 18:51:32.792554 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.792524 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09742c1d-7e12-40c1-95bf-f889a52be85e-config-file\") pod \"09742c1d-7e12-40c1-95bf-f889a52be85e\" (UID: \"09742c1d-7e12-40c1-95bf-f889a52be85e\") " Jul 24 18:51:32.792967 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.792935 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09742c1d-7e12-40c1-95bf-f889a52be85e-config-file" (OuterVolumeSpecName: "config-file") pod "09742c1d-7e12-40c1-95bf-f889a52be85e" (UID: "09742c1d-7e12-40c1-95bf-f889a52be85e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:51:32.794735 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.794710 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09742c1d-7e12-40c1-95bf-f889a52be85e-kube-api-access-ddnvg" (OuterVolumeSpecName: "kube-api-access-ddnvg") pod "09742c1d-7e12-40c1-95bf-f889a52be85e" (UID: "09742c1d-7e12-40c1-95bf-f889a52be85e"). InnerVolumeSpecName "kube-api-access-ddnvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:51:32.863652 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.863619 2575 generic.go:356] "Generic (PLEG): container finished" podID="09742c1d-7e12-40c1-95bf-f889a52be85e" containerID="d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058" exitCode=0 Jul 24 18:51:32.863785 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.863669 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" Jul 24 18:51:32.863785 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.863703 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" event={"ID":"09742c1d-7e12-40c1-95bf-f889a52be85e","Type":"ContainerDied","Data":"d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058"} Jul 24 18:51:32.863785 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.863773 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6rwkq" event={"ID":"09742c1d-7e12-40c1-95bf-f889a52be85e","Type":"ContainerDied","Data":"aa9c0f7ab1cb0d6e5c35206d25d6b5035679ffb830f78910fa7b033321f3b043"} Jul 24 18:51:32.863894 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.863798 2575 scope.go:117] "RemoveContainer" containerID="d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058" Jul 24 18:51:32.872603 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.872586 2575 scope.go:117] "RemoveContainer" containerID="d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058" Jul 24 18:51:32.872903 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:51:32.872885 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058\": container with ID starting with d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058 not found: ID does not exist" containerID="d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058" Jul 24 18:51:32.872973 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.872914 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058"} err="failed to get container status \"d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058\": rpc error: code = NotFound desc = could not find container \"d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058\": container with ID starting with d0b14ed25fd6e29a440c93307131339f002dffe6b205ee0248483f9af3ab1058 not found: ID does not exist" Jul 24 18:51:32.887569 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.885224 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6rwkq"] Jul 24 18:51:32.892481 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.892460 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6rwkq"] Jul 24 18:51:32.893252 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.893236 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddnvg\" (UniqueName: \"kubernetes.io/projected/09742c1d-7e12-40c1-95bf-f889a52be85e-kube-api-access-ddnvg\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:51:32.893299 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:32.893257 2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09742c1d-7e12-40c1-95bf-f889a52be85e-config-file\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:51:33.417626 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.417593 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09742c1d-7e12-40c1-95bf-f889a52be85e" path="/var/lib/kubelet/pods/09742c1d-7e12-40c1-95bf-f889a52be85e/volumes" Jul 24 18:51:33.808202 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.808123 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-99zqr"] Jul 24 18:51:33.808549 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.808533 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09742c1d-7e12-40c1-95bf-f889a52be85e" containerName="limitador" Jul 24 18:51:33.808591 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.808553 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="09742c1d-7e12-40c1-95bf-f889a52be85e" containerName="limitador" Jul 24 18:51:33.808639 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.808620 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="09742c1d-7e12-40c1-95bf-f889a52be85e" containerName="limitador" Jul 24 18:51:33.812819 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.812800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" Jul 24 18:51:33.815516 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.815494 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 18:51:33.815710 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.815501 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:51:33.815710 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.815570 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 18:51:33.816549 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.816532 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wvglz\"" Jul 24 18:51:33.820096 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.820041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-99zqr"] Jul 24 18:51:33.902882 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.902851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc85abbe-1231-4fe3-a47d-9aff88e08c22-data\") pod \"postgres-69fb4594cf-99zqr\" (UID: \"bc85abbe-1231-4fe3-a47d-9aff88e08c22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" Jul 24 18:51:33.902882 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:33.902888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d89nh\" (UniqueName: \"kubernetes.io/projected/bc85abbe-1231-4fe3-a47d-9aff88e08c22-kube-api-access-d89nh\") pod \"postgres-69fb4594cf-99zqr\" (UID: \"bc85abbe-1231-4fe3-a47d-9aff88e08c22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" Jul 24 18:51:34.004121 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:34.004083 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d89nh\" (UniqueName: \"kubernetes.io/projected/bc85abbe-1231-4fe3-a47d-9aff88e08c22-kube-api-access-d89nh\") pod \"postgres-69fb4594cf-99zqr\" (UID: \"bc85abbe-1231-4fe3-a47d-9aff88e08c22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" Jul 24 18:51:34.004249 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:34.004175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc85abbe-1231-4fe3-a47d-9aff88e08c22-data\") pod \"postgres-69fb4594cf-99zqr\" (UID: \"bc85abbe-1231-4fe3-a47d-9aff88e08c22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" Jul 24 18:51:34.004490 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:34.004475 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc85abbe-1231-4fe3-a47d-9aff88e08c22-data\") pod \"postgres-69fb4594cf-99zqr\" (UID: \"bc85abbe-1231-4fe3-a47d-9aff88e08c22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" Jul 24 18:51:34.012585 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:34.012556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d89nh\" (UniqueName: \"kubernetes.io/projected/bc85abbe-1231-4fe3-a47d-9aff88e08c22-kube-api-access-d89nh\") pod \"postgres-69fb4594cf-99zqr\" (UID: \"bc85abbe-1231-4fe3-a47d-9aff88e08c22\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" Jul 24 18:51:34.124277 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:34.124202 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" Jul 24 18:51:34.258256 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:34.258140 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-99zqr"] Jul 24 18:51:34.261022 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:51:34.260985 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc85abbe_1231_4fe3_a47d_9aff88e08c22.slice/crio-583ee27102a975f23c7f78f88830d770bcfca31efb043fd531289080ea8395c1 WatchSource:0}: Error finding container 583ee27102a975f23c7f78f88830d770bcfca31efb043fd531289080ea8395c1: Status 404 returned error can't find the container with id 583ee27102a975f23c7f78f88830d770bcfca31efb043fd531289080ea8395c1 Jul 24 18:51:34.875307 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:34.875273 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" event={"ID":"bc85abbe-1231-4fe3-a47d-9aff88e08c22","Type":"ContainerStarted","Data":"583ee27102a975f23c7f78f88830d770bcfca31efb043fd531289080ea8395c1"} Jul 24 18:51:39.902942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:39.902905 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" event={"ID":"bc85abbe-1231-4fe3-a47d-9aff88e08c22","Type":"ContainerStarted","Data":"61d65fa656220cd5521775e901f188428d0f2f98af8ec19ca0690fe55637b895"} Jul 24 18:51:39.903308 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:39.902971 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" Jul 24 18:51:39.919521 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:39.919458 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" podStartSLOduration=2.17638212 podStartE2EDuration="6.91943962s" totalImagesPullingTime="4.7430575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:51:39.918695966 +0000 UTC m=+494.995208523" watchObservedRunningTime="2026-07-24 18:51:39.91943962 +0000 UTC m=+494.995952178" Jul 24 18:51:45.936934 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:45.936895 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-99zqr" Jul 24 18:51:54.093931 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.093893 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fc77b6c75-wh7sg"] Jul 24 18:51:54.121487 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.121452 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fc77b6c75-wh7sg"] Jul 24 18:51:54.121722 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.121528 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:51:54.125755 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.125569 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5gxt4\"" Jul 24 18:51:54.125755 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.125575 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 18:51:54.125755 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.125579 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 18:51:54.280545 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.280511 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-t8kd9"] Jul 24 18:51:54.282883 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.282852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-webhook-cert\") pod \"maas-controller-6fc77b6c75-wh7sg\" (UID: \"2c190a05-1a6c-4b3c-91eb-ece2398a94c0\") " pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:51:54.283027 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.283011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfbz\" (UniqueName: \"kubernetes.io/projected/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-kube-api-access-2nfbz\") pod \"maas-controller-6fc77b6c75-wh7sg\" (UID: \"2c190a05-1a6c-4b3c-91eb-ece2398a94c0\") " pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:51:54.284135 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.284119 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:51:54.293334 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.293312 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-t8kd9"] Jul 24 18:51:54.383975 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.383897 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfbz\" (UniqueName: \"kubernetes.io/projected/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-kube-api-access-2nfbz\") pod \"maas-controller-6fc77b6c75-wh7sg\" (UID: \"2c190a05-1a6c-4b3c-91eb-ece2398a94c0\") " pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:51:54.383975 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.383948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-webhook-cert\") pod \"maas-controller-6fc77b6c75-wh7sg\" (UID: \"2c190a05-1a6c-4b3c-91eb-ece2398a94c0\") " pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:51:54.384154 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.383983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cec4b11b-ea69-46cd-8704-b3104b6c8646-webhook-cert\") pod \"maas-controller-54c89c5bc7-t8kd9\" (UID: \"cec4b11b-ea69-46cd-8704-b3104b6c8646\") " pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:51:54.384154 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.384021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp8zz\" (UniqueName: \"kubernetes.io/projected/cec4b11b-ea69-46cd-8704-b3104b6c8646-kube-api-access-qp8zz\") pod \"maas-controller-54c89c5bc7-t8kd9\" (UID: \"cec4b11b-ea69-46cd-8704-b3104b6c8646\") " pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:51:54.386554 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.386534 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-webhook-cert\") pod \"maas-controller-6fc77b6c75-wh7sg\" (UID: \"2c190a05-1a6c-4b3c-91eb-ece2398a94c0\") " pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:51:54.392235 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.392211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfbz\" (UniqueName: \"kubernetes.io/projected/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-kube-api-access-2nfbz\") pod \"maas-controller-6fc77b6c75-wh7sg\" (UID: \"2c190a05-1a6c-4b3c-91eb-ece2398a94c0\") " pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:51:54.434841 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.434819 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:51:54.485724 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.485688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp8zz\" (UniqueName: \"kubernetes.io/projected/cec4b11b-ea69-46cd-8704-b3104b6c8646-kube-api-access-qp8zz\") pod \"maas-controller-54c89c5bc7-t8kd9\" (UID: \"cec4b11b-ea69-46cd-8704-b3104b6c8646\") " pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:51:54.485921 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.485901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cec4b11b-ea69-46cd-8704-b3104b6c8646-webhook-cert\") pod \"maas-controller-54c89c5bc7-t8kd9\" (UID: \"cec4b11b-ea69-46cd-8704-b3104b6c8646\") " pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:51:54.488618 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.488593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cec4b11b-ea69-46cd-8704-b3104b6c8646-webhook-cert\") pod \"maas-controller-54c89c5bc7-t8kd9\" (UID: \"cec4b11b-ea69-46cd-8704-b3104b6c8646\") " pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:51:54.495024 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.494999 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp8zz\" (UniqueName: \"kubernetes.io/projected/cec4b11b-ea69-46cd-8704-b3104b6c8646-kube-api-access-qp8zz\") pod \"maas-controller-54c89c5bc7-t8kd9\" (UID: \"cec4b11b-ea69-46cd-8704-b3104b6c8646\") " pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:51:54.572088 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.572066 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fc77b6c75-wh7sg"] Jul 24 18:51:54.574177 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:51:54.574152 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c190a05_1a6c_4b3c_91eb_ece2398a94c0.slice/crio-eca6de5b89b26359cdbc9e0ff424a7d6705a95b68ec8fed8c0866b558d06ed60 WatchSource:0}: Error finding container eca6de5b89b26359cdbc9e0ff424a7d6705a95b68ec8fed8c0866b558d06ed60: Status 404 returned error can't find the container with id eca6de5b89b26359cdbc9e0ff424a7d6705a95b68ec8fed8c0866b558d06ed60 Jul 24 18:51:54.594945 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.594919 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:51:54.732435 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.732411 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-t8kd9"] Jul 24 18:51:54.734293 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:51:54.734267 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec4b11b_ea69_46cd_8704_b3104b6c8646.slice/crio-8870b4caa077f17e10ea817e71f7c54d6e47d2f9c65110849b4b250e687add49 WatchSource:0}: Error finding container 8870b4caa077f17e10ea817e71f7c54d6e47d2f9c65110849b4b250e687add49: Status 404 returned error can't find the container with id 8870b4caa077f17e10ea817e71f7c54d6e47d2f9c65110849b4b250e687add49 Jul 24 18:51:54.975807 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.975693 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" event={"ID":"cec4b11b-ea69-46cd-8704-b3104b6c8646","Type":"ContainerStarted","Data":"8870b4caa077f17e10ea817e71f7c54d6e47d2f9c65110849b4b250e687add49"} Jul 24 18:51:54.977154 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:54.977093 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" event={"ID":"2c190a05-1a6c-4b3c-91eb-ece2398a94c0","Type":"ContainerStarted","Data":"eca6de5b89b26359cdbc9e0ff424a7d6705a95b68ec8fed8c0866b558d06ed60"} Jul 24 18:51:56.996718 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:56.996697 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:51:57.014833 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:57.014773 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" podStartSLOduration=0.669331443 podStartE2EDuration="3.01473403s" totalImagesPullingTime="2.345402587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:51:57.014337388 +0000 UTC m=+512.090849948" watchObservedRunningTime="2026-07-24 18:51:57.01473403 +0000 UTC m=+512.091246589" Jul 24 18:51:58.004372 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:58.004330 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" event={"ID":"2c190a05-1a6c-4b3c-91eb-ece2398a94c0","Type":"ContainerStarted","Data":"7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549"} Jul 24 18:51:58.006364 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:58.006338 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" event={"ID":"cec4b11b-ea69-46cd-8704-b3104b6c8646","Type":"ContainerStarted","Data":"4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620"} Jul 24 18:51:58.006500 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:58.006466 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:51:58.024188 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:51:58.024136 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" podStartSLOduration=1.8325393700000001 podStartE2EDuration="4.024121839s" totalImagesPullingTime="2.191582469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:51:58.023216618 +0000 UTC m=+513.099729195" watchObservedRunningTime="2026-07-24 18:51:58.024121839 +0000 UTC m=+513.100634397" Jul 24 18:52:09.013538 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.013503 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:52:09.013997 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.013585 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:52:09.075752 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.075717 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fc77b6c75-wh7sg"] Jul 24 18:52:09.075985 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.075944 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" podUID="2c190a05-1a6c-4b3c-91eb-ece2398a94c0" containerName="manager" containerID="cri-o://7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549" gracePeriod=30 Jul 24 18:52:09.333919 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.333892 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:52:09.417571 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.417539 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nfbz\" (UniqueName: \"kubernetes.io/projected/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-kube-api-access-2nfbz\") pod \"2c190a05-1a6c-4b3c-91eb-ece2398a94c0\" (UID: \"2c190a05-1a6c-4b3c-91eb-ece2398a94c0\") " Jul 24 18:52:09.417780 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.417601 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-webhook-cert\") pod \"2c190a05-1a6c-4b3c-91eb-ece2398a94c0\" (UID: \"2c190a05-1a6c-4b3c-91eb-ece2398a94c0\") " Jul 24 18:52:09.419960 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.419930 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2c190a05-1a6c-4b3c-91eb-ece2398a94c0" (UID: "2c190a05-1a6c-4b3c-91eb-ece2398a94c0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:52:09.419960 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.419944 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-kube-api-access-2nfbz" (OuterVolumeSpecName: "kube-api-access-2nfbz") pod "2c190a05-1a6c-4b3c-91eb-ece2398a94c0" (UID: "2c190a05-1a6c-4b3c-91eb-ece2398a94c0"). InnerVolumeSpecName "kube-api-access-2nfbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:52:09.519248 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.519217 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-webhook-cert\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:52:09.519248 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:09.519245 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nfbz\" (UniqueName: \"kubernetes.io/projected/2c190a05-1a6c-4b3c-91eb-ece2398a94c0-kube-api-access-2nfbz\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:52:10.067302 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:10.067271 2575 generic.go:356] "Generic (PLEG): container finished" podID="2c190a05-1a6c-4b3c-91eb-ece2398a94c0" containerID="7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549" exitCode=0 Jul 24 18:52:10.067302 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:10.067307 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" event={"ID":"2c190a05-1a6c-4b3c-91eb-ece2398a94c0","Type":"ContainerDied","Data":"7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549"} Jul 24 18:52:10.067829 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:10.067317 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" Jul 24 18:52:10.067829 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:10.067345 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fc77b6c75-wh7sg" event={"ID":"2c190a05-1a6c-4b3c-91eb-ece2398a94c0","Type":"ContainerDied","Data":"eca6de5b89b26359cdbc9e0ff424a7d6705a95b68ec8fed8c0866b558d06ed60"} Jul 24 18:52:10.067829 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:10.067368 2575 scope.go:117] "RemoveContainer" containerID="7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549" Jul 24 18:52:10.076716 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:10.076528 2575 scope.go:117] "RemoveContainer" containerID="7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549" Jul 24 18:52:10.076808 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:52:10.076781 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549\": container with ID starting with 7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549 not found: ID does not exist" containerID="7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549" Jul 24 18:52:10.076846 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:10.076809 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549"} err="failed to get container status \"7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549\": rpc error: code = NotFound desc = could not find container \"7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549\": container with ID starting with 7af457f61d94843ce91b757153e1ad93453314127cfc4081e9ccb04947476549 not found: ID does not exist" Jul 24 18:52:10.088476 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:10.088454 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fc77b6c75-wh7sg"] Jul 24 18:52:10.094872 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:10.094853 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6fc77b6c75-wh7sg"] Jul 24 18:52:11.417083 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:11.417049 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c190a05-1a6c-4b3c-91eb-ece2398a94c0" path="/var/lib/kubelet/pods/2c190a05-1a6c-4b3c-91eb-ece2398a94c0/volumes" Jul 24 18:52:27.117630 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.117588 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs"] Jul 24 18:52:27.118170 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.118152 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c190a05-1a6c-4b3c-91eb-ece2398a94c0" containerName="manager" Jul 24 18:52:27.118249 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.118174 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c190a05-1a6c-4b3c-91eb-ece2398a94c0" containerName="manager" Jul 24 18:52:27.118315 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.118302 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c190a05-1a6c-4b3c-91eb-ece2398a94c0" containerName="manager" Jul 24 18:52:27.122918 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.122895 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.125563 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.125535 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 18:52:27.125664 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.125651 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cvjqn\"" Jul 24 18:52:27.131571 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.131548 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs"] Jul 24 18:52:27.273251 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.273206 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23891aef-a3ee-4728-9f4d-5edb97010bec-openshift-service-ca\") pod \"maas-api-6f5d8786fb-4hrxs\" (UID: \"23891aef-a3ee-4728-9f4d-5edb97010bec\") " pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.273440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.273285 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23891aef-a3ee-4728-9f4d-5edb97010bec-maas-api-tls\") pod \"maas-api-6f5d8786fb-4hrxs\" (UID: \"23891aef-a3ee-4728-9f4d-5edb97010bec\") " pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.273440 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.273319 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4874v\" (UniqueName: \"kubernetes.io/projected/23891aef-a3ee-4728-9f4d-5edb97010bec-kube-api-access-4874v\") pod \"maas-api-6f5d8786fb-4hrxs\" (UID: \"23891aef-a3ee-4728-9f4d-5edb97010bec\") " pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.374763 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.374661 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23891aef-a3ee-4728-9f4d-5edb97010bec-openshift-service-ca\") pod \"maas-api-6f5d8786fb-4hrxs\" (UID: \"23891aef-a3ee-4728-9f4d-5edb97010bec\") " pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.374916 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.374777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23891aef-a3ee-4728-9f4d-5edb97010bec-maas-api-tls\") pod \"maas-api-6f5d8786fb-4hrxs\" (UID: \"23891aef-a3ee-4728-9f4d-5edb97010bec\") " pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.374916 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.374830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4874v\" (UniqueName: \"kubernetes.io/projected/23891aef-a3ee-4728-9f4d-5edb97010bec-kube-api-access-4874v\") pod \"maas-api-6f5d8786fb-4hrxs\" (UID: \"23891aef-a3ee-4728-9f4d-5edb97010bec\") " pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.375311 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.375289 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23891aef-a3ee-4728-9f4d-5edb97010bec-openshift-service-ca\") pod \"maas-api-6f5d8786fb-4hrxs\" (UID: \"23891aef-a3ee-4728-9f4d-5edb97010bec\") " pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.377293 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.377276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23891aef-a3ee-4728-9f4d-5edb97010bec-maas-api-tls\") pod \"maas-api-6f5d8786fb-4hrxs\" (UID: \"23891aef-a3ee-4728-9f4d-5edb97010bec\") " pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.383069 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.383046 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4874v\" (UniqueName: \"kubernetes.io/projected/23891aef-a3ee-4728-9f4d-5edb97010bec-kube-api-access-4874v\") pod \"maas-api-6f5d8786fb-4hrxs\" (UID: \"23891aef-a3ee-4728-9f4d-5edb97010bec\") " pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.435071 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.435046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:27.581619 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.581593 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs"] Jul 24 18:52:27.583943 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:52:27.583917 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23891aef_a3ee_4728_9f4d_5edb97010bec.slice/crio-0cad58576b89b2e8a8a8b5f6cc2b744ae7e9b55a6d9a20dbd32ad7871e1d25a6 WatchSource:0}: Error finding container 0cad58576b89b2e8a8a8b5f6cc2b744ae7e9b55a6d9a20dbd32ad7871e1d25a6: Status 404 returned error can't find the container with id 0cad58576b89b2e8a8a8b5f6cc2b744ae7e9b55a6d9a20dbd32ad7871e1d25a6 Jul 24 18:52:27.672836 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.672798 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q"] Jul 24 18:52:27.677585 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.677562 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" Jul 24 18:52:27.680294 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.680273 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5c8mj\"" Jul 24 18:52:27.680404 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.680314 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 18:52:27.689093 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.689072 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q"] Jul 24 18:52:27.705683 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.705658 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vqv6d"] Jul 24 18:52:27.709171 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.709150 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" Jul 24 18:52:27.723394 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.723372 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vqv6d"] Jul 24 18:52:27.779179 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.779153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dktdl\" (UniqueName: \"kubernetes.io/projected/be404140-206a-4cb9-b1e2-191c44c91c7f-kube-api-access-dktdl\") pod \"payload-processing-5cc6cd444b-vqv6d\" (UID: \"be404140-206a-4cb9-b1e2-191c44c91c7f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" Jul 24 18:52:27.779346 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.779186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be404140-206a-4cb9-b1e2-191c44c91c7f-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vqv6d\" (UID: \"be404140-206a-4cb9-b1e2-191c44c91c7f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" Jul 24 18:52:27.779346 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.779310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmdn\" (UniqueName: \"kubernetes.io/projected/408c6a82-efa6-4c5d-99be-66f3fb7efdd7-kube-api-access-qgmdn\") pod \"payload-pre-processing-687dbb6cf6-wjp2q\" (UID: \"408c6a82-efa6-4c5d-99be-66f3fb7efdd7\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" Jul 24 18:52:27.779346 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.779341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408c6a82-efa6-4c5d-99be-66f3fb7efdd7-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wjp2q\" (UID: \"408c6a82-efa6-4c5d-99be-66f3fb7efdd7\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" Jul 24 18:52:27.880163 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.880131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmdn\" (UniqueName: \"kubernetes.io/projected/408c6a82-efa6-4c5d-99be-66f3fb7efdd7-kube-api-access-qgmdn\") pod \"payload-pre-processing-687dbb6cf6-wjp2q\" (UID: \"408c6a82-efa6-4c5d-99be-66f3fb7efdd7\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" Jul 24 18:52:27.880163 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.880169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408c6a82-efa6-4c5d-99be-66f3fb7efdd7-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wjp2q\" (UID: \"408c6a82-efa6-4c5d-99be-66f3fb7efdd7\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" Jul 24 18:52:27.880432 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.880190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dktdl\" (UniqueName: \"kubernetes.io/projected/be404140-206a-4cb9-b1e2-191c44c91c7f-kube-api-access-dktdl\") pod \"payload-processing-5cc6cd444b-vqv6d\" (UID: \"be404140-206a-4cb9-b1e2-191c44c91c7f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" Jul 24 18:52:27.880432 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.880207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be404140-206a-4cb9-b1e2-191c44c91c7f-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vqv6d\" (UID: \"be404140-206a-4cb9-b1e2-191c44c91c7f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" Jul 24 18:52:27.880961 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.880936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be404140-206a-4cb9-b1e2-191c44c91c7f-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vqv6d\" (UID: \"be404140-206a-4cb9-b1e2-191c44c91c7f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" Jul 24 18:52:27.881067 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.880988 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/408c6a82-efa6-4c5d-99be-66f3fb7efdd7-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wjp2q\" (UID: \"408c6a82-efa6-4c5d-99be-66f3fb7efdd7\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" Jul 24 18:52:27.888621 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.888599 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmdn\" (UniqueName: \"kubernetes.io/projected/408c6a82-efa6-4c5d-99be-66f3fb7efdd7-kube-api-access-qgmdn\") pod \"payload-pre-processing-687dbb6cf6-wjp2q\" (UID: \"408c6a82-efa6-4c5d-99be-66f3fb7efdd7\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" Jul 24 18:52:27.888703 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.888646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dktdl\" (UniqueName: \"kubernetes.io/projected/be404140-206a-4cb9-b1e2-191c44c91c7f-kube-api-access-dktdl\") pod \"payload-processing-5cc6cd444b-vqv6d\" (UID: \"be404140-206a-4cb9-b1e2-191c44c91c7f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" Jul 24 18:52:27.988840 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:27.988733 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" Jul 24 18:52:28.019696 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:28.019670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" Jul 24 18:52:28.147422 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:28.146672 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q"] Jul 24 18:52:28.149015 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:52:28.148985 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408c6a82_efa6_4c5d_99be_66f3fb7efdd7.slice/crio-745b15a287f19994712b57f701ca05003fe82bff1edaf5db72509cc6ab6ccda8 WatchSource:0}: Error finding container 745b15a287f19994712b57f701ca05003fe82bff1edaf5db72509cc6ab6ccda8: Status 404 returned error can't find the container with id 745b15a287f19994712b57f701ca05003fe82bff1edaf5db72509cc6ab6ccda8 Jul 24 18:52:28.164361 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:28.164326 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" event={"ID":"23891aef-a3ee-4728-9f4d-5edb97010bec","Type":"ContainerStarted","Data":"0cad58576b89b2e8a8a8b5f6cc2b744ae7e9b55a6d9a20dbd32ad7871e1d25a6"} Jul 24 18:52:28.165672 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:28.165642 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" event={"ID":"408c6a82-efa6-4c5d-99be-66f3fb7efdd7","Type":"ContainerStarted","Data":"745b15a287f19994712b57f701ca05003fe82bff1edaf5db72509cc6ab6ccda8"} Jul 24 18:52:28.181191 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:28.181133 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vqv6d"] Jul 24 18:52:28.183138 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:52:28.183115 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe404140_206a_4cb9_b1e2_191c44c91c7f.slice/crio-f483ffe81eeac9ce06ad9de76f1f952247e2b491a1c89d5c00482c8d3bbdda78 WatchSource:0}: Error finding container f483ffe81eeac9ce06ad9de76f1f952247e2b491a1c89d5c00482c8d3bbdda78: Status 404 returned error can't find the container with id f483ffe81eeac9ce06ad9de76f1f952247e2b491a1c89d5c00482c8d3bbdda78 Jul 24 18:52:29.178964 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:29.178923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" event={"ID":"be404140-206a-4cb9-b1e2-191c44c91c7f","Type":"ContainerStarted","Data":"f483ffe81eeac9ce06ad9de76f1f952247e2b491a1c89d5c00482c8d3bbdda78"} Jul 24 18:52:30.186557 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:30.186515 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" event={"ID":"23891aef-a3ee-4728-9f4d-5edb97010bec","Type":"ContainerStarted","Data":"817ad6c618e161d870536e0ffeb690ddeab386960e60e2bb2b2c3cad853e870f"} Jul 24 18:52:30.187024 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:30.186641 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:30.205468 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:30.205351 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" podStartSLOduration=1.293626994 podStartE2EDuration="3.205330423s" totalImagesPullingTime="1.911703429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:52:30.203864004 +0000 UTC m=+545.280376563" watchObservedRunningTime="2026-07-24 18:52:30.205330423 +0000 UTC m=+545.281842984" Jul 24 18:52:31.193334 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:31.193295 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" event={"ID":"408c6a82-efa6-4c5d-99be-66f3fb7efdd7","Type":"ContainerStarted","Data":"5913a53851bcdc9e2253ec36cb9a819752567987918b0d37c08b6a231e14066c"} Jul 24 18:52:31.193829 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:31.193494 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" Jul 24 18:52:31.194830 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:31.194804 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" event={"ID":"be404140-206a-4cb9-b1e2-191c44c91c7f","Type":"ContainerStarted","Data":"f0276f85ee1ba48e30332648242e08afa9216d520b687878a0dd04bdc289eeb4"} Jul 24 18:52:31.194955 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:31.194862 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" Jul 24 18:52:31.209625 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:31.209582 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" podStartSLOduration=1.636781577 podStartE2EDuration="4.209568724s" totalImagesPullingTime="2.572787147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:52:31.208188757 +0000 UTC m=+546.284701351" watchObservedRunningTime="2026-07-24 18:52:31.209568724 +0000 UTC m=+546.286081281" Jul 24 18:52:31.223091 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:31.223050 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" podStartSLOduration=1.677011293 podStartE2EDuration="4.223040203s" totalImagesPullingTime="2.54602891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:52:31.222051692 +0000 UTC m=+546.298564250" watchObservedRunningTime="2026-07-24 18:52:31.223040203 +0000 UTC m=+546.299552761" Jul 24 18:52:36.200571 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:36.200538 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f5d8786fb-4hrxs" Jul 24 18:52:42.203386 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:42.203351 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-vqv6d" Jul 24 18:52:42.203801 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:42.203431 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wjp2q" Jul 24 18:52:44.459476 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.459438 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h"] Jul 24 18:52:44.465605 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.465583 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.468673 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.468650 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 18:52:44.470004 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.469974 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 18:52:44.470139 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.469989 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 24 18:52:44.470139 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.470128 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tnr97\"" Jul 24 18:52:44.473050 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.472558 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h"] Jul 24 18:52:44.528931 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.528902 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58b7a06-38bb-442a-940c-58a647c44d28-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.528931 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.528934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.529111 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.528957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.529111 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.528978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjmt\" (UniqueName: \"kubernetes.io/projected/f58b7a06-38bb-442a-940c-58a647c44d28-kube-api-access-jqjmt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.529111 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.529055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.529210 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.529108 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.629588 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.629558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58b7a06-38bb-442a-940c-58a647c44d28-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.629588 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.629594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.629876 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.629616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.629876 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.629646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjmt\" (UniqueName: \"kubernetes.io/projected/f58b7a06-38bb-442a-940c-58a647c44d28-kube-api-access-jqjmt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.629876 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.629677 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.629876 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.629713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.630164 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.630140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.630309 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.630286 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.630592 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.630317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.632265 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.632238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f58b7a06-38bb-442a-940c-58a647c44d28-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.632355 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.632320 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f58b7a06-38bb-442a-940c-58a647c44d28-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.639545 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.639518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjmt\" (UniqueName: \"kubernetes.io/projected/f58b7a06-38bb-442a-940c-58a647c44d28-kube-api-access-jqjmt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h\" (UID: \"f58b7a06-38bb-442a-940c-58a647c44d28\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.777811 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.777697 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:44.926851 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:44.926708 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h"] Jul 24 18:52:44.929194 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:52:44.929169 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58b7a06_38bb_442a_940c_58a647c44d28.slice/crio-93c65fbc877fc6753cf8221c2e32cbfdb2808ffcd69ba73634cff4a8de177c9b WatchSource:0}: Error finding container 93c65fbc877fc6753cf8221c2e32cbfdb2808ffcd69ba73634cff4a8de177c9b: Status 404 returned error can't find the container with id 93c65fbc877fc6753cf8221c2e32cbfdb2808ffcd69ba73634cff4a8de177c9b Jul 24 18:52:45.271664 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:45.271628 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" event={"ID":"f58b7a06-38bb-442a-940c-58a647c44d28","Type":"ContainerStarted","Data":"93c65fbc877fc6753cf8221c2e32cbfdb2808ffcd69ba73634cff4a8de177c9b"} Jul 24 18:52:46.278382 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:46.278302 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" event={"ID":"f58b7a06-38bb-442a-940c-58a647c44d28","Type":"ContainerStarted","Data":"345a56493ea356babe9b23f8f801e8d68b67b1af2fd6a6147a157cda50189e61"} Jul 24 18:52:46.299654 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:46.299605 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" podStartSLOduration=1.299220759 podStartE2EDuration="2.299586714s" totalImagesPullingTime="1.000365955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:52:46.298451352 +0000 UTC m=+561.374963931" watchObservedRunningTime="2026-07-24 18:52:46.299586714 +0000 UTC m=+561.376099272" Jul 24 18:52:54.778354 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:54.778306 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:54.778837 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:54.778380 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:54.791918 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:54.791886 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:52:55.338009 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:52:55.337980 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h" Jul 24 18:53:11.373471 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.373430 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc"] Jul 24 18:53:11.379047 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.379023 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.381879 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.381858 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 24 18:53:11.390060 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.390033 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc"] Jul 24 18:53:11.480575 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.480534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vphtw\" (UniqueName: \"kubernetes.io/projected/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-kube-api-access-vphtw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.480758 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.480663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.480758 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.480726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.480853 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.480838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.480898 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.480884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.480933 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.480913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.581625 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.581591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.581625 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.581629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.581892 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.581658 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.581892 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.581696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vphtw\" (UniqueName: \"kubernetes.io/projected/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-kube-api-access-vphtw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.581892 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.581790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.581892 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.581814 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.582102 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.582040 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.582159 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.582098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.582212 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.582193 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.584181 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.584154 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.584408 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.584390 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.589947 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.589925 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vphtw\" (UniqueName: \"kubernetes.io/projected/ef53f9fb-b3e8-4339-b2a6-3dfba7892130-kube-api-access-vphtw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8p9qc\" (UID: \"ef53f9fb-b3e8-4339-b2a6-3dfba7892130\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.692190 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.692144 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:11.836718 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:11.836691 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc"] Jul 24 18:53:11.838578 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:53:11.838543 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef53f9fb_b3e8_4339_b2a6_3dfba7892130.slice/crio-ab9bac305b6778cda42a3a1014d3904e60464bd8c965d4ea3c7d2eeb818d5f18 WatchSource:0}: Error finding container ab9bac305b6778cda42a3a1014d3904e60464bd8c965d4ea3c7d2eeb818d5f18: Status 404 returned error can't find the container with id ab9bac305b6778cda42a3a1014d3904e60464bd8c965d4ea3c7d2eeb818d5f18 Jul 24 18:53:12.417991 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:12.417955 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" event={"ID":"ef53f9fb-b3e8-4339-b2a6-3dfba7892130","Type":"ContainerStarted","Data":"04fafc412e62da0b6bd2d884e00a4aef0506cdc34d1e620f6819e8099661db9d"} Jul 24 18:53:12.417991 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:12.417997 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" event={"ID":"ef53f9fb-b3e8-4339-b2a6-3dfba7892130","Type":"ContainerStarted","Data":"ab9bac305b6778cda42a3a1014d3904e60464bd8c965d4ea3c7d2eeb818d5f18"} Jul 24 18:53:12.437271 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:12.437217 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" podStartSLOduration=1.197884768 podStartE2EDuration="1.437201816s" totalImagesPullingTime="239.317048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:53:12.435688427 +0000 UTC m=+587.512200997" watchObservedRunningTime="2026-07-24 18:53:12.437201816 +0000 UTC m=+587.513714419" Jul 24 18:53:21.692578 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:21.692541 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:21.693040 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:21.692604 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:21.705391 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:21.705366 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:22.488376 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:22.488350 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8p9qc" Jul 24 18:53:25.377082 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:25.377057 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 18:53:25.384963 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:25.384940 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 18:53:25.386422 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:25.386400 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 18:53:25.395205 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:53:25.395185 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 18:54:51.876477 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:51.876396 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-t8kd9"] Jul 24 18:54:51.877182 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:51.876648 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" podUID="cec4b11b-ea69-46cd-8704-b3104b6c8646" containerName="manager" containerID="cri-o://4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620" gracePeriod=30 Jul 24 18:54:52.133120 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.133057 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:54:52.273028 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.273001 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp8zz\" (UniqueName: \"kubernetes.io/projected/cec4b11b-ea69-46cd-8704-b3104b6c8646-kube-api-access-qp8zz\") pod \"cec4b11b-ea69-46cd-8704-b3104b6c8646\" (UID: \"cec4b11b-ea69-46cd-8704-b3104b6c8646\") " Jul 24 18:54:52.273190 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.273096 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cec4b11b-ea69-46cd-8704-b3104b6c8646-webhook-cert\") pod \"cec4b11b-ea69-46cd-8704-b3104b6c8646\" (UID: \"cec4b11b-ea69-46cd-8704-b3104b6c8646\") " Jul 24 18:54:52.275204 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.275165 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cec4b11b-ea69-46cd-8704-b3104b6c8646-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cec4b11b-ea69-46cd-8704-b3104b6c8646" (UID: "cec4b11b-ea69-46cd-8704-b3104b6c8646"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:54:52.275323 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.275221 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec4b11b-ea69-46cd-8704-b3104b6c8646-kube-api-access-qp8zz" (OuterVolumeSpecName: "kube-api-access-qp8zz") pod "cec4b11b-ea69-46cd-8704-b3104b6c8646" (UID: "cec4b11b-ea69-46cd-8704-b3104b6c8646"). InnerVolumeSpecName "kube-api-access-qp8zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:54:52.374026 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.373995 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cec4b11b-ea69-46cd-8704-b3104b6c8646-webhook-cert\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:54:52.374026 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.374020 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp8zz\" (UniqueName: \"kubernetes.io/projected/cec4b11b-ea69-46cd-8704-b3104b6c8646-kube-api-access-qp8zz\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:54:52.937517 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.937480 2575 generic.go:356] "Generic (PLEG): container finished" podID="cec4b11b-ea69-46cd-8704-b3104b6c8646" containerID="4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620" exitCode=0 Jul 24 18:54:52.937953 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.937530 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" Jul 24 18:54:52.937953 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.937566 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" event={"ID":"cec4b11b-ea69-46cd-8704-b3104b6c8646","Type":"ContainerDied","Data":"4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620"} Jul 24 18:54:52.937953 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.937622 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c89c5bc7-t8kd9" event={"ID":"cec4b11b-ea69-46cd-8704-b3104b6c8646","Type":"ContainerDied","Data":"8870b4caa077f17e10ea817e71f7c54d6e47d2f9c65110849b4b250e687add49"} Jul 24 18:54:52.937953 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.937645 2575 scope.go:117] "RemoveContainer" containerID="4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620" Jul 24 18:54:52.947668 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.947645 2575 scope.go:117] "RemoveContainer" containerID="4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620" Jul 24 18:54:52.947967 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:54:52.947942 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620\": container with ID starting with 4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620 not found: ID does not exist" containerID="4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620" Jul 24 18:54:52.948043 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.947971 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620"} err="failed to get container status \"4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620\": rpc error: code = NotFound desc = could not find container \"4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620\": container with ID starting with 4cac588dd92387c3b6ca6850340e5fd057c0cfcaeea48319286cbeaa7f37a620 not found: ID does not exist" Jul 24 18:54:52.973187 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.973159 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-t8kd9"] Jul 24 18:54:52.995481 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:52.995446 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-t8kd9"] Jul 24 18:54:53.417507 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:54:53.417475 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cec4b11b-ea69-46cd-8704-b3104b6c8646" path="/var/lib/kubelet/pods/cec4b11b-ea69-46cd-8704-b3104b6c8646/volumes" Jul 24 18:55:53.808590 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.808555 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-h4lws"] Jul 24 18:55:53.809122 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.808993 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cec4b11b-ea69-46cd-8704-b3104b6c8646" containerName="manager" Jul 24 18:55:53.809122 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.809006 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cec4b11b-ea69-46cd-8704-b3104b6c8646" containerName="manager" Jul 24 18:55:53.809122 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.809081 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cec4b11b-ea69-46cd-8704-b3104b6c8646" containerName="manager" Jul 24 18:55:53.812075 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.812060 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:55:53.815634 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.815613 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5gxt4\"" Jul 24 18:55:53.815786 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.815636 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 18:55:53.815786 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.815671 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 18:55:53.822069 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.822041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-h4lws"] Jul 24 18:55:53.939149 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.939115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkgh\" (UniqueName: \"kubernetes.io/projected/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-kube-api-access-6gkgh\") pod \"maas-controller-54c89c5bc7-h4lws\" (UID: \"68667bbe-8d79-4c71-a771-c7b8c25d2fa0\") " pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:55:53.939320 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:53.939160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-webhook-cert\") pod \"maas-controller-54c89c5bc7-h4lws\" (UID: \"68667bbe-8d79-4c71-a771-c7b8c25d2fa0\") " pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:55:54.040462 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:54.040415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkgh\" (UniqueName: \"kubernetes.io/projected/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-kube-api-access-6gkgh\") pod \"maas-controller-54c89c5bc7-h4lws\" (UID: \"68667bbe-8d79-4c71-a771-c7b8c25d2fa0\") " pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:55:54.040658 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:54.040487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-webhook-cert\") pod \"maas-controller-54c89c5bc7-h4lws\" (UID: \"68667bbe-8d79-4c71-a771-c7b8c25d2fa0\") " pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:55:54.043101 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:54.043082 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-webhook-cert\") pod \"maas-controller-54c89c5bc7-h4lws\" (UID: \"68667bbe-8d79-4c71-a771-c7b8c25d2fa0\") " pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:55:54.049252 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:54.049230 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkgh\" (UniqueName: \"kubernetes.io/projected/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-kube-api-access-6gkgh\") pod \"maas-controller-54c89c5bc7-h4lws\" (UID: \"68667bbe-8d79-4c71-a771-c7b8c25d2fa0\") " pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:55:54.123355 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:54.123272 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:55:54.267400 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:54.267378 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-h4lws"] Jul 24 18:55:54.269194 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:55:54.269170 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68667bbe_8d79_4c71_a771_c7b8c25d2fa0.slice/crio-4f946b56063a37c6d9476de45017735f5a968535bd4015b95e2531fc7f621692 WatchSource:0}: Error finding container 4f946b56063a37c6d9476de45017735f5a968535bd4015b95e2531fc7f621692: Status 404 returned error can't find the container with id 4f946b56063a37c6d9476de45017735f5a968535bd4015b95e2531fc7f621692 Jul 24 18:55:54.270532 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:54.270510 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 18:55:55.258170 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:55.258138 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" event={"ID":"68667bbe-8d79-4c71-a771-c7b8c25d2fa0","Type":"ContainerStarted","Data":"ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd"} Jul 24 18:55:55.258542 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:55.258194 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" event={"ID":"68667bbe-8d79-4c71-a771-c7b8c25d2fa0","Type":"ContainerStarted","Data":"4f946b56063a37c6d9476de45017735f5a968535bd4015b95e2531fc7f621692"} Jul 24 18:55:55.258542 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:55.258269 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:55:55.277213 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:55:55.277160 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" podStartSLOduration=1.75720018 podStartE2EDuration="2.277144815s" totalImagesPullingTime="519.944635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:55:55.274919893 +0000 UTC m=+750.351432451" watchObservedRunningTime="2026-07-24 18:55:55.277144815 +0000 UTC m=+750.353657373" Jul 24 18:56:06.266333 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:06.266300 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:56:10.900586 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:10.900555 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-h4lws"] Jul 24 18:56:10.901052 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:10.900735 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" podUID="68667bbe-8d79-4c71-a771-c7b8c25d2fa0" containerName="manager" containerID="cri-o://ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd" gracePeriod=30 Jul 24 18:56:11.159337 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.159281 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:56:11.187585 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.187556 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gkgh\" (UniqueName: \"kubernetes.io/projected/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-kube-api-access-6gkgh\") pod \"68667bbe-8d79-4c71-a771-c7b8c25d2fa0\" (UID: \"68667bbe-8d79-4c71-a771-c7b8c25d2fa0\") " Jul 24 18:56:11.187788 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.187596 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-webhook-cert\") pod \"68667bbe-8d79-4c71-a771-c7b8c25d2fa0\" (UID: \"68667bbe-8d79-4c71-a771-c7b8c25d2fa0\") " Jul 24 18:56:11.189787 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.189757 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "68667bbe-8d79-4c71-a771-c7b8c25d2fa0" (UID: "68667bbe-8d79-4c71-a771-c7b8c25d2fa0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:56:11.189916 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.189887 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-kube-api-access-6gkgh" (OuterVolumeSpecName: "kube-api-access-6gkgh") pod "68667bbe-8d79-4c71-a771-c7b8c25d2fa0" (UID: "68667bbe-8d79-4c71-a771-c7b8c25d2fa0"). InnerVolumeSpecName "kube-api-access-6gkgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:56:11.288703 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.288670 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gkgh\" (UniqueName: \"kubernetes.io/projected/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-kube-api-access-6gkgh\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:56:11.288703 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.288696 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68667bbe-8d79-4c71-a771-c7b8c25d2fa0-webhook-cert\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 18:56:11.346262 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.346227 2575 generic.go:356] "Generic (PLEG): container finished" podID="68667bbe-8d79-4c71-a771-c7b8c25d2fa0" containerID="ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd" exitCode=0 Jul 24 18:56:11.346459 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.346280 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" Jul 24 18:56:11.346459 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.346305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" event={"ID":"68667bbe-8d79-4c71-a771-c7b8c25d2fa0","Type":"ContainerDied","Data":"ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd"} Jul 24 18:56:11.346459 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.346360 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c89c5bc7-h4lws" event={"ID":"68667bbe-8d79-4c71-a771-c7b8c25d2fa0","Type":"ContainerDied","Data":"4f946b56063a37c6d9476de45017735f5a968535bd4015b95e2531fc7f621692"} Jul 24 18:56:11.346459 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.346389 2575 scope.go:117] "RemoveContainer" containerID="ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd" Jul 24 18:56:11.356190 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.356171 2575 scope.go:117] "RemoveContainer" containerID="ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd" Jul 24 18:56:11.356444 ip-10-0-134-30 kubenswrapper[2575]: E0724 18:56:11.356425 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd\": container with ID starting with ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd not found: ID does not exist" containerID="ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd" Jul 24 18:56:11.356483 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.356456 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd"} err="failed to get container status \"ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd\": rpc error: code = NotFound desc = could not find container \"ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd\": container with ID starting with ed062088b17788c8075dcc267bac234f687d67c147031a1fa9ee612db02bd5bd not found: ID does not exist" Jul 24 18:56:11.368307 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.368281 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-h4lws"] Jul 24 18:56:11.375250 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.375213 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-h4lws"] Jul 24 18:56:11.416880 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:11.416797 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68667bbe-8d79-4c71-a771-c7b8c25d2fa0" path="/var/lib/kubelet/pods/68667bbe-8d79-4c71-a771-c7b8c25d2fa0/volumes" Jul 24 18:56:12.139563 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.139531 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-bpn4j"] Jul 24 18:56:12.139976 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.139962 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68667bbe-8d79-4c71-a771-c7b8c25d2fa0" containerName="manager" Jul 24 18:56:12.139976 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.139976 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="68667bbe-8d79-4c71-a771-c7b8c25d2fa0" containerName="manager" Jul 24 18:56:12.140058 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.140050 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="68667bbe-8d79-4c71-a771-c7b8c25d2fa0" containerName="manager" Jul 24 18:56:12.144677 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.144657 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" Jul 24 18:56:12.147446 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.147425 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 18:56:12.148479 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.148459 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 18:56:12.148544 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.148500 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5gxt4\"" Jul 24 18:56:12.154729 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.154708 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-bpn4j"] Jul 24 18:56:12.195706 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.195667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/425ac1be-3355-4c91-8e9c-06e526b7e2e9-webhook-cert\") pod \"maas-controller-54c89c5bc7-bpn4j\" (UID: \"425ac1be-3355-4c91-8e9c-06e526b7e2e9\") " pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" Jul 24 18:56:12.195888 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.195717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshzh\" (UniqueName: \"kubernetes.io/projected/425ac1be-3355-4c91-8e9c-06e526b7e2e9-kube-api-access-kshzh\") pod \"maas-controller-54c89c5bc7-bpn4j\" (UID: \"425ac1be-3355-4c91-8e9c-06e526b7e2e9\") " pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" Jul 24 18:56:12.297132 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.297100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/425ac1be-3355-4c91-8e9c-06e526b7e2e9-webhook-cert\") pod \"maas-controller-54c89c5bc7-bpn4j\" (UID: \"425ac1be-3355-4c91-8e9c-06e526b7e2e9\") " pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" Jul 24 18:56:12.297314 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.297143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshzh\" (UniqueName: \"kubernetes.io/projected/425ac1be-3355-4c91-8e9c-06e526b7e2e9-kube-api-access-kshzh\") pod \"maas-controller-54c89c5bc7-bpn4j\" (UID: \"425ac1be-3355-4c91-8e9c-06e526b7e2e9\") " pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" Jul 24 18:56:12.299967 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.299939 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/425ac1be-3355-4c91-8e9c-06e526b7e2e9-webhook-cert\") pod \"maas-controller-54c89c5bc7-bpn4j\" (UID: \"425ac1be-3355-4c91-8e9c-06e526b7e2e9\") " pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" Jul 24 18:56:12.305805 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.305778 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshzh\" (UniqueName: \"kubernetes.io/projected/425ac1be-3355-4c91-8e9c-06e526b7e2e9-kube-api-access-kshzh\") pod \"maas-controller-54c89c5bc7-bpn4j\" (UID: \"425ac1be-3355-4c91-8e9c-06e526b7e2e9\") " pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" Jul 24 18:56:12.455425 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.455335 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" Jul 24 18:56:12.596914 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:12.596845 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54c89c5bc7-bpn4j"] Jul 24 18:56:12.599911 ip-10-0-134-30 kubenswrapper[2575]: W0724 18:56:12.599881 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425ac1be_3355_4c91_8e9c_06e526b7e2e9.slice/crio-ccdfd9020422e8654c4fc24085a06fbf14da6adbd36dd8be07a8abec0f447f06 WatchSource:0}: Error finding container ccdfd9020422e8654c4fc24085a06fbf14da6adbd36dd8be07a8abec0f447f06: Status 404 returned error can't find the container with id ccdfd9020422e8654c4fc24085a06fbf14da6adbd36dd8be07a8abec0f447f06 Jul 24 18:56:13.357942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:13.357840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" event={"ID":"425ac1be-3355-4c91-8e9c-06e526b7e2e9","Type":"ContainerStarted","Data":"03cfd75833ed9aa89e25307b012f1bb63bf4a72041e274f4e3f6434b2e02893d"} Jul 24 18:56:13.357942 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:13.357906 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" event={"ID":"425ac1be-3355-4c91-8e9c-06e526b7e2e9","Type":"ContainerStarted","Data":"ccdfd9020422e8654c4fc24085a06fbf14da6adbd36dd8be07a8abec0f447f06"} Jul 24 18:56:13.358535 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:13.358021 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" Jul 24 18:56:13.375319 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:13.375269 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" podStartSLOduration=1.05075557 podStartE2EDuration="1.375255098s" totalImagesPullingTime="324.499528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:56:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:56:13.374352966 +0000 UTC m=+768.450865524" watchObservedRunningTime="2026-07-24 18:56:13.375255098 +0000 UTC m=+768.451767656" Jul 24 18:56:24.365390 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:56:24.365356 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54c89c5bc7-bpn4j" Jul 24 18:58:25.411520 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:58:25.411429 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 18:58:25.420355 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:58:25.420329 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 18:58:25.421090 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:58:25.421070 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 18:58:25.429610 ip-10-0-134-30 kubenswrapper[2575]: I0724 18:58:25.429591 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:00:00.143457 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:00.143378 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748660-9tbbg"] Jul 24 19:00:00.147029 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:00.147011 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748660-9tbbg" Jul 24 19:00:00.150543 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:00.150524 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:00:00.156474 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:00.156458 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748660-9tbbg" Jul 24 19:00:00.166137 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:00.166114 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748660-9tbbg"] Jul 24 19:00:00.507089 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:00.507005 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748660-9tbbg"] Jul 24 19:00:00.510201 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:00:00.510173 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c4967b_eda3_40b4_bd98_2c69d5582d4c.slice/crio-ddac697d23712767423e6baf1bbf022d2137af32694143f0d767dad580cbbe96 WatchSource:0}: Error finding container ddac697d23712767423e6baf1bbf022d2137af32694143f0d767dad580cbbe96: Status 404 returned error can't find the container with id ddac697d23712767423e6baf1bbf022d2137af32694143f0d767dad580cbbe96 Jul 24 19:00:00.545608 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:00.545572 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748660-9tbbg" event={"ID":"37c4967b-eda3-40b4-bd98-2c69d5582d4c","Type":"ContainerStarted","Data":"ddac697d23712767423e6baf1bbf022d2137af32694143f0d767dad580cbbe96"} Jul 24 19:00:04.568956 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:04.568923 2575 generic.go:356] "Generic (PLEG): container finished" podID="37c4967b-eda3-40b4-bd98-2c69d5582d4c" containerID="52440bc16389d1fbd63fa6e614a07746bd175dd03909143c189467374b44b451" exitCode=0 Jul 24 19:00:04.569359 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:04.569017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748660-9tbbg" event={"ID":"37c4967b-eda3-40b4-bd98-2c69d5582d4c","Type":"ContainerDied","Data":"52440bc16389d1fbd63fa6e614a07746bd175dd03909143c189467374b44b451"} Jul 24 19:00:05.720567 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:05.720543 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748660-9tbbg" Jul 24 19:00:06.581781 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:06.581733 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748660-9tbbg" event={"ID":"37c4967b-eda3-40b4-bd98-2c69d5582d4c","Type":"ContainerDied","Data":"ddac697d23712767423e6baf1bbf022d2137af32694143f0d767dad580cbbe96"} Jul 24 19:00:06.581781 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:06.581779 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748660-9tbbg" Jul 24 19:00:06.581995 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:00:06.581796 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddac697d23712767423e6baf1bbf022d2137af32694143f0d767dad580cbbe96" Jul 24 19:01:08.744517 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:08.744482 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4"] Jul 24 19:01:08.745067 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:08.744682 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" podUID="4b2cc625-0c89-4443-8346-7c6362cbbcb3" containerName="manager" containerID="cri-o://f40b0d632a58f3be69563642a651b40f1813f361c0e5cce8fc8362204a623ab5" gracePeriod=10 Jul 24 19:01:11.930040 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:11.930007 2575 generic.go:356] "Generic (PLEG): container finished" podID="4b2cc625-0c89-4443-8346-7c6362cbbcb3" containerID="f40b0d632a58f3be69563642a651b40f1813f361c0e5cce8fc8362204a623ab5" exitCode=0 Jul 24 19:01:11.930418 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:11.930085 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" event={"ID":"4b2cc625-0c89-4443-8346-7c6362cbbcb3","Type":"ContainerDied","Data":"f40b0d632a58f3be69563642a651b40f1813f361c0e5cce8fc8362204a623ab5"} Jul 24 19:01:12.107654 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.107631 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 19:01:12.189555 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.189475 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw9hj\" (UniqueName: \"kubernetes.io/projected/4b2cc625-0c89-4443-8346-7c6362cbbcb3-kube-api-access-gw9hj\") pod \"4b2cc625-0c89-4443-8346-7c6362cbbcb3\" (UID: \"4b2cc625-0c89-4443-8346-7c6362cbbcb3\") " Jul 24 19:01:12.189555 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.189517 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b2cc625-0c89-4443-8346-7c6362cbbcb3-extensions-socket-volume\") pod \"4b2cc625-0c89-4443-8346-7c6362cbbcb3\" (UID: \"4b2cc625-0c89-4443-8346-7c6362cbbcb3\") " Jul 24 19:01:12.189893 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.189869 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b2cc625-0c89-4443-8346-7c6362cbbcb3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4b2cc625-0c89-4443-8346-7c6362cbbcb3" (UID: "4b2cc625-0c89-4443-8346-7c6362cbbcb3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 19:01:12.191702 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.191678 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2cc625-0c89-4443-8346-7c6362cbbcb3-kube-api-access-gw9hj" (OuterVolumeSpecName: "kube-api-access-gw9hj") pod "4b2cc625-0c89-4443-8346-7c6362cbbcb3" (UID: "4b2cc625-0c89-4443-8346-7c6362cbbcb3"). InnerVolumeSpecName "kube-api-access-gw9hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:01:12.290835 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.290802 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw9hj\" (UniqueName: \"kubernetes.io/projected/4b2cc625-0c89-4443-8346-7c6362cbbcb3-kube-api-access-gw9hj\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:01:12.290835 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.290828 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b2cc625-0c89-4443-8346-7c6362cbbcb3-extensions-socket-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:01:12.936156 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.936127 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" Jul 24 19:01:12.936586 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.936128 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4" event={"ID":"4b2cc625-0c89-4443-8346-7c6362cbbcb3","Type":"ContainerDied","Data":"a226c5ddb868370ce498621446de51c1a8a94dfb3a8ad2c4823e4038767ae043"} Jul 24 19:01:12.936586 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.936260 2575 scope.go:117] "RemoveContainer" containerID="f40b0d632a58f3be69563642a651b40f1813f361c0e5cce8fc8362204a623ab5" Jul 24 19:01:12.958812 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.958788 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4"] Jul 24 19:01:12.962489 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:12.962465 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l42t4"] Jul 24 19:01:13.417128 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:01:13.417096 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2cc625-0c89-4443-8346-7c6362cbbcb3" path="/var/lib/kubelet/pods/4b2cc625-0c89-4443-8346-7c6362cbbcb3/volumes" Jul 24 19:03:10.875206 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.875116 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k"] Jul 24 19:03:10.875827 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.875760 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37c4967b-eda3-40b4-bd98-2c69d5582d4c" containerName="cleanup" Jul 24 19:03:10.875827 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.875780 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="37c4967b-eda3-40b4-bd98-2c69d5582d4c" containerName="cleanup" Jul 24 19:03:10.875827 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.875803 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2cc625-0c89-4443-8346-7c6362cbbcb3" containerName="manager" Jul 24 19:03:10.875827 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.875812 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2cc625-0c89-4443-8346-7c6362cbbcb3" containerName="manager" Jul 24 19:03:10.876054 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.875948 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="37c4967b-eda3-40b4-bd98-2c69d5582d4c" containerName="cleanup" Jul 24 19:03:10.876054 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.875966 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2cc625-0c89-4443-8346-7c6362cbbcb3" containerName="manager" Jul 24 19:03:10.879074 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.879054 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" Jul 24 19:03:10.881896 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.881877 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f7xtj\"" Jul 24 19:03:10.895048 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.895019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68qp\" (UniqueName: \"kubernetes.io/projected/4a98bc24-4210-4e6e-af0b-4b45b943c9c1-kube-api-access-r68qp\") pod \"kuadrant-operator-controller-manager-86878c769f-lqz4k\" (UID: \"4a98bc24-4210-4e6e-af0b-4b45b943c9c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" Jul 24 19:03:10.895148 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.895058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a98bc24-4210-4e6e-af0b-4b45b943c9c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lqz4k\" (UID: \"4a98bc24-4210-4e6e-af0b-4b45b943c9c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" Jul 24 19:03:10.902178 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.902158 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k"] Jul 24 19:03:10.996295 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.996257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68qp\" (UniqueName: \"kubernetes.io/projected/4a98bc24-4210-4e6e-af0b-4b45b943c9c1-kube-api-access-r68qp\") pod \"kuadrant-operator-controller-manager-86878c769f-lqz4k\" (UID: \"4a98bc24-4210-4e6e-af0b-4b45b943c9c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" Jul 24 19:03:10.996485 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.996307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a98bc24-4210-4e6e-af0b-4b45b943c9c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lqz4k\" (UID: \"4a98bc24-4210-4e6e-af0b-4b45b943c9c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" Jul 24 19:03:10.996655 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:10.996639 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a98bc24-4210-4e6e-af0b-4b45b943c9c1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lqz4k\" (UID: \"4a98bc24-4210-4e6e-af0b-4b45b943c9c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" Jul 24 19:03:11.010020 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:11.009987 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68qp\" (UniqueName: \"kubernetes.io/projected/4a98bc24-4210-4e6e-af0b-4b45b943c9c1-kube-api-access-r68qp\") pod \"kuadrant-operator-controller-manager-86878c769f-lqz4k\" (UID: \"4a98bc24-4210-4e6e-af0b-4b45b943c9c1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" Jul 24 19:03:11.189952 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:11.189922 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" Jul 24 19:03:11.340176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:11.340148 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k"] Jul 24 19:03:11.343500 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:03:11.343452 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a98bc24_4210_4e6e_af0b_4b45b943c9c1.slice/crio-b6b848d27d9ad20827286743d7846b5db0a30c255a5b0c54b10fea10ca2b21fc WatchSource:0}: Error finding container b6b848d27d9ad20827286743d7846b5db0a30c255a5b0c54b10fea10ca2b21fc: Status 404 returned error can't find the container with id b6b848d27d9ad20827286743d7846b5db0a30c255a5b0c54b10fea10ca2b21fc Jul 24 19:03:11.541952 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:11.541858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" event={"ID":"4a98bc24-4210-4e6e-af0b-4b45b943c9c1","Type":"ContainerStarted","Data":"24ff7dc6cf1310c74eaf1c9f8eb0f9148b1d8cccb50763ef6b82b8c8a41898a0"} Jul 24 19:03:11.541952 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:11.541908 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" event={"ID":"4a98bc24-4210-4e6e-af0b-4b45b943c9c1","Type":"ContainerStarted","Data":"b6b848d27d9ad20827286743d7846b5db0a30c255a5b0c54b10fea10ca2b21fc"} Jul 24 19:03:11.542147 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:11.541998 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" Jul 24 19:03:11.560578 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:11.560525 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" podStartSLOduration=1.560507806 podStartE2EDuration="1.560507806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:03:11.559006478 +0000 UTC m=+1186.635519036" watchObservedRunningTime="2026-07-24 19:03:11.560507806 +0000 UTC m=+1186.637020364" Jul 24 19:03:22.549723 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:22.549694 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lqz4k" Jul 24 19:03:25.445647 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:25.445622 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:03:25.452215 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:25.452195 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:03:25.453296 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:25.453280 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:03:25.463620 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:03:25.463599 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:08:25.490367 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:08:25.490329 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:08:25.495304 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:08:25.495281 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:08:25.499177 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:08:25.499146 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:08:25.504448 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:08:25.504429 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:09:52.916392 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:52.916353 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54"] Jul 24 19:09:52.920182 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:52.920161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:09:52.925294 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:52.925273 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d198c699\"" Jul 24 19:09:52.925402 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:52.925281 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d198c699-dockercfg-wxt58\"" Jul 24 19:09:52.931830 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:52.931807 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54"] Jul 24 19:09:53.003069 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.003035 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9dp\" (UniqueName: \"kubernetes.io/projected/dbdefdd2-f98a-4d70-ac01-6360a1db6208-kube-api-access-gt9dp\") pod \"payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54\" (UID: \"dbdefdd2-f98a-4d70-ac01-6360a1db6208\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:09:53.003228 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.003101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdefdd2-f98a-4d70-ac01-6360a1db6208-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54\" (UID: \"dbdefdd2-f98a-4d70-ac01-6360a1db6208\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:09:53.103587 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.103545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9dp\" (UniqueName: \"kubernetes.io/projected/dbdefdd2-f98a-4d70-ac01-6360a1db6208-kube-api-access-gt9dp\") pod \"payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54\" (UID: \"dbdefdd2-f98a-4d70-ac01-6360a1db6208\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:09:53.103846 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.103618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdefdd2-f98a-4d70-ac01-6360a1db6208-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54\" (UID: \"dbdefdd2-f98a-4d70-ac01-6360a1db6208\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:09:53.104402 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.104372 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdefdd2-f98a-4d70-ac01-6360a1db6208-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54\" (UID: \"dbdefdd2-f98a-4d70-ac01-6360a1db6208\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:09:53.112387 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.112363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9dp\" (UniqueName: \"kubernetes.io/projected/dbdefdd2-f98a-4d70-ac01-6360a1db6208-kube-api-access-gt9dp\") pod \"payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54\" (UID: \"dbdefdd2-f98a-4d70-ac01-6360a1db6208\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:09:53.231055 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.230959 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:09:53.582186 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.582159 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54"] Jul 24 19:09:53.584210 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:09:53.584179 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdefdd2_f98a_4d70_ac01_6360a1db6208.slice/crio-69a9bc28ae1619a88a0a2c86549404d7bace50d7b2d46c21668d7f19e7923f80 WatchSource:0}: Error finding container 69a9bc28ae1619a88a0a2c86549404d7bace50d7b2d46c21668d7f19e7923f80: Status 404 returned error can't find the container with id 69a9bc28ae1619a88a0a2c86549404d7bace50d7b2d46c21668d7f19e7923f80 Jul 24 19:09:53.684092 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.684059 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" event={"ID":"dbdefdd2-f98a-4d70-ac01-6360a1db6208","Type":"ContainerStarted","Data":"1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760"} Jul 24 19:09:53.684236 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.684115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" event={"ID":"dbdefdd2-f98a-4d70-ac01-6360a1db6208","Type":"ContainerStarted","Data":"69a9bc28ae1619a88a0a2c86549404d7bace50d7b2d46c21668d7f19e7923f80"} Jul 24 19:09:53.684236 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.684141 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:09:53.700630 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:09:53.700584 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" podStartSLOduration=1.7005673909999999 podStartE2EDuration="1.700567391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:09:53.699385408 +0000 UTC m=+1588.775897993" watchObservedRunningTime="2026-07-24 19:09:53.700567391 +0000 UTC m=+1588.777079949" Jul 24 19:10:04.691944 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:04.691911 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:10:17.515640 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:17.515551 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw"] Jul 24 19:10:17.519635 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:17.519611 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw" Jul 24 19:10:17.522489 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:17.522468 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:10:17.532124 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:17.532097 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw"] Jul 24 19:10:17.615386 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:17.615357 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376c6e5f-4e65-446b-a63d-2e0b6e704779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d198c699-44pvw\" (UID: \"376c6e5f-4e65-446b-a63d-2e0b6e704779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw" Jul 24 19:10:17.716383 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:17.716338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376c6e5f-4e65-446b-a63d-2e0b6e704779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d198c699-44pvw\" (UID: \"376c6e5f-4e65-446b-a63d-2e0b6e704779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw" Jul 24 19:10:17.717041 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:17.717014 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376c6e5f-4e65-446b-a63d-2e0b6e704779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d198c699-44pvw\" (UID: \"376c6e5f-4e65-446b-a63d-2e0b6e704779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw" Jul 24 19:10:17.829299 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:17.829210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw" Jul 24 19:10:17.967928 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:17.967905 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw"] Jul 24 19:10:17.970396 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:10:17.970367 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376c6e5f_4e65_446b_a63d_2e0b6e704779.slice/crio-3ee1f0ac71cc0900ecfdd1093354fe6c6a5c193791cf90e2f8debb24565e986e WatchSource:0}: Error finding container 3ee1f0ac71cc0900ecfdd1093354fe6c6a5c193791cf90e2f8debb24565e986e: Status 404 returned error can't find the container with id 3ee1f0ac71cc0900ecfdd1093354fe6c6a5c193791cf90e2f8debb24565e986e Jul 24 19:10:18.816649 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:18.816611 2575 generic.go:356] "Generic (PLEG): container finished" podID="376c6e5f-4e65-446b-a63d-2e0b6e704779" containerID="0e35a63d3d58db722863e9d8c8315d2c4bdf5f8a09f7ea551b03e17444da9ab6" exitCode=0 Jul 24 19:10:18.817042 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:18.816699 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw" event={"ID":"376c6e5f-4e65-446b-a63d-2e0b6e704779","Type":"ContainerDied","Data":"0e35a63d3d58db722863e9d8c8315d2c4bdf5f8a09f7ea551b03e17444da9ab6"} Jul 24 19:10:18.817042 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:18.816772 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw" event={"ID":"376c6e5f-4e65-446b-a63d-2e0b6e704779","Type":"ContainerStarted","Data":"3ee1f0ac71cc0900ecfdd1093354fe6c6a5c193791cf90e2f8debb24565e986e"} Jul 24 19:10:19.968896 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:19.968870 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw" Jul 24 19:10:20.037986 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:20.037944 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376c6e5f-4e65-446b-a63d-2e0b6e704779-maas-api-service-ca\") pod \"376c6e5f-4e65-446b-a63d-2e0b6e704779\" (UID: \"376c6e5f-4e65-446b-a63d-2e0b6e704779\") " Jul 24 19:10:20.038335 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:20.038315 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376c6e5f-4e65-446b-a63d-2e0b6e704779-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "376c6e5f-4e65-446b-a63d-2e0b6e704779" (UID: "376c6e5f-4e65-446b-a63d-2e0b6e704779"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:10:20.139026 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:20.138998 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/376c6e5f-4e65-446b-a63d-2e0b6e704779-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:10:20.829009 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:20.828980 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw" Jul 24 19:10:20.829009 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:20.828992 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw" event={"ID":"376c6e5f-4e65-446b-a63d-2e0b6e704779","Type":"ContainerDied","Data":"3ee1f0ac71cc0900ecfdd1093354fe6c6a5c193791cf90e2f8debb24565e986e"} Jul 24 19:10:20.829213 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:20.829041 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ee1f0ac71cc0900ecfdd1093354fe6c6a5c193791cf90e2f8debb24565e986e" Jul 24 19:10:28.182438 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.182405 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54"] Jul 24 19:10:28.183073 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.182622 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" podUID="dbdefdd2-f98a-4d70-ac01-6360a1db6208" containerName="payload-pre-processing" containerID="cri-o://1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760" gracePeriod=30 Jul 24 19:10:28.453497 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.453468 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:10:28.515136 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.515107 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdefdd2-f98a-4d70-ac01-6360a1db6208-plugins-config-volume\") pod \"dbdefdd2-f98a-4d70-ac01-6360a1db6208\" (UID: \"dbdefdd2-f98a-4d70-ac01-6360a1db6208\") " Jul 24 19:10:28.515311 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.515230 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt9dp\" (UniqueName: \"kubernetes.io/projected/dbdefdd2-f98a-4d70-ac01-6360a1db6208-kube-api-access-gt9dp\") pod \"dbdefdd2-f98a-4d70-ac01-6360a1db6208\" (UID: \"dbdefdd2-f98a-4d70-ac01-6360a1db6208\") " Jul 24 19:10:28.515533 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.515506 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdefdd2-f98a-4d70-ac01-6360a1db6208-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dbdefdd2-f98a-4d70-ac01-6360a1db6208" (UID: "dbdefdd2-f98a-4d70-ac01-6360a1db6208"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:10:28.517416 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.517394 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdefdd2-f98a-4d70-ac01-6360a1db6208-kube-api-access-gt9dp" (OuterVolumeSpecName: "kube-api-access-gt9dp") pod "dbdefdd2-f98a-4d70-ac01-6360a1db6208" (UID: "dbdefdd2-f98a-4d70-ac01-6360a1db6208"). InnerVolumeSpecName "kube-api-access-gt9dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:10:28.616933 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.616903 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdefdd2-f98a-4d70-ac01-6360a1db6208-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:10:28.616933 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.616926 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9dp\" (UniqueName: \"kubernetes.io/projected/dbdefdd2-f98a-4d70-ac01-6360a1db6208-kube-api-access-gt9dp\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:10:28.870669 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.870590 2575 generic.go:356] "Generic (PLEG): container finished" podID="dbdefdd2-f98a-4d70-ac01-6360a1db6208" containerID="1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760" exitCode=0 Jul 24 19:10:28.870669 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.870639 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" Jul 24 19:10:28.870895 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.870673 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" event={"ID":"dbdefdd2-f98a-4d70-ac01-6360a1db6208","Type":"ContainerDied","Data":"1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760"} Jul 24 19:10:28.870895 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.870731 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54" event={"ID":"dbdefdd2-f98a-4d70-ac01-6360a1db6208","Type":"ContainerDied","Data":"69a9bc28ae1619a88a0a2c86549404d7bace50d7b2d46c21668d7f19e7923f80"} Jul 24 19:10:28.870895 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.870775 2575 scope.go:117] "RemoveContainer" containerID="1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760" Jul 24 19:10:28.881995 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.881971 2575 scope.go:117] "RemoveContainer" containerID="1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760" Jul 24 19:10:28.882306 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:10:28.882282 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760\": container with ID starting with 1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760 not found: ID does not exist" containerID="1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760" Jul 24 19:10:28.882422 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.882313 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760"} err="failed to get container status \"1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760\": rpc error: code = NotFound desc = could not find container \"1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760\": container with ID starting with 1551f35b1fa02b034c042441019a893463e6943337e413cd30178ffce16a8760 not found: ID does not exist" Jul 24 19:10:28.892139 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.892113 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54"] Jul 24 19:10:28.896654 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:28.896635 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d198c699-75ddc6979b-bzj54"] Jul 24 19:10:29.417391 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:29.417355 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbdefdd2-f98a-4d70-ac01-6360a1db6208" path="/var/lib/kubelet/pods/dbdefdd2-f98a-4d70-ac01-6360a1db6208/volumes" Jul 24 19:10:32.603171 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:32.603121 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw"] Jul 24 19:10:32.605533 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:32.605509 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d198c699-44pvw"] Jul 24 19:10:33.417799 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:10:33.417762 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="376c6e5f-4e65-446b-a63d-2e0b6e704779" path="/var/lib/kubelet/pods/376c6e5f-4e65-446b-a63d-2e0b6e704779/volumes" Jul 24 19:11:04.033445 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.033404 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf"] Jul 24 19:11:04.034111 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.034092 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="376c6e5f-4e65-446b-a63d-2e0b6e704779" containerName="revoke-keys" Jul 24 19:11:04.034162 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.034115 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="376c6e5f-4e65-446b-a63d-2e0b6e704779" containerName="revoke-keys" Jul 24 19:11:04.034162 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.034152 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdefdd2-f98a-4d70-ac01-6360a1db6208" containerName="payload-pre-processing" Jul 24 19:11:04.034226 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.034161 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdefdd2-f98a-4d70-ac01-6360a1db6208" containerName="payload-pre-processing" Jul 24 19:11:04.034260 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.034248 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdefdd2-f98a-4d70-ac01-6360a1db6208" containerName="payload-pre-processing" Jul 24 19:11:04.034293 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.034261 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="376c6e5f-4e65-446b-a63d-2e0b6e704779" containerName="revoke-keys" Jul 24 19:11:04.037898 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.037879 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" Jul 24 19:11:04.040668 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.040634 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:11:04.042821 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.042796 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf"] Jul 24 19:11:04.122871 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.122838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2427b70-8f2e-42e8-9235-f80670cc01f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf\" (UID: \"e2427b70-8f2e-42e8-9235-f80670cc01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" Jul 24 19:11:04.224375 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.224337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2427b70-8f2e-42e8-9235-f80670cc01f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf\" (UID: \"e2427b70-8f2e-42e8-9235-f80670cc01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" Jul 24 19:11:04.225015 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.224991 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2427b70-8f2e-42e8-9235-f80670cc01f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf\" (UID: \"e2427b70-8f2e-42e8-9235-f80670cc01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" Jul 24 19:11:04.349892 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.349800 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" Jul 24 19:11:04.488657 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:04.488630 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf"] Jul 24 19:11:04.491087 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:11:04.491057 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2427b70_8f2e_42e8_9235_f80670cc01f8.slice/crio-dd2a3a63107176e83b481e13a62c68b72cb18fd0a4ada2cf2054bbcf243d780a WatchSource:0}: Error finding container dd2a3a63107176e83b481e13a62c68b72cb18fd0a4ada2cf2054bbcf243d780a: Status 404 returned error can't find the container with id dd2a3a63107176e83b481e13a62c68b72cb18fd0a4ada2cf2054bbcf243d780a Jul 24 19:11:05.071421 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:05.071328 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" event={"ID":"e2427b70-8f2e-42e8-9235-f80670cc01f8","Type":"ContainerStarted","Data":"14110f780cd87f4b3c897794c2bd05c55be37b85feff1da9ac462faed8bbcd40"} Jul 24 19:11:05.071421 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:05.071381 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" event={"ID":"e2427b70-8f2e-42e8-9235-f80670cc01f8","Type":"ContainerStarted","Data":"dd2a3a63107176e83b481e13a62c68b72cb18fd0a4ada2cf2054bbcf243d780a"} Jul 24 19:11:05.088246 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:05.088197 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" podStartSLOduration=1.088182525 podStartE2EDuration="1.088182525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:11:05.086867803 +0000 UTC m=+1660.163380361" watchObservedRunningTime="2026-07-24 19:11:05.088182525 +0000 UTC m=+1660.164695105" Jul 24 19:11:06.077492 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:06.077457 2575 generic.go:356] "Generic (PLEG): container finished" podID="e2427b70-8f2e-42e8-9235-f80670cc01f8" containerID="14110f780cd87f4b3c897794c2bd05c55be37b85feff1da9ac462faed8bbcd40" exitCode=0 Jul 24 19:11:06.077971 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:06.077540 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" event={"ID":"e2427b70-8f2e-42e8-9235-f80670cc01f8","Type":"ContainerDied","Data":"14110f780cd87f4b3c897794c2bd05c55be37b85feff1da9ac462faed8bbcd40"} Jul 24 19:11:07.221689 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:07.221661 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" Jul 24 19:11:07.350465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:07.350381 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2427b70-8f2e-42e8-9235-f80670cc01f8-maas-api-service-ca\") pod \"e2427b70-8f2e-42e8-9235-f80670cc01f8\" (UID: \"e2427b70-8f2e-42e8-9235-f80670cc01f8\") " Jul 24 19:11:07.350730 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:07.350708 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2427b70-8f2e-42e8-9235-f80670cc01f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2427b70-8f2e-42e8-9235-f80670cc01f8" (UID: "e2427b70-8f2e-42e8-9235-f80670cc01f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:11:07.451662 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:07.451632 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2427b70-8f2e-42e8-9235-f80670cc01f8-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:11:08.088276 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:08.088251 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" Jul 24 19:11:08.088502 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:08.088247 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf" event={"ID":"e2427b70-8f2e-42e8-9235-f80670cc01f8","Type":"ContainerDied","Data":"dd2a3a63107176e83b481e13a62c68b72cb18fd0a4ada2cf2054bbcf243d780a"} Jul 24 19:11:08.088502 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:08.088375 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd2a3a63107176e83b481e13a62c68b72cb18fd0a4ada2cf2054bbcf243d780a" Jul 24 19:11:19.119656 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:19.119619 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf"] Jul 24 19:11:19.121657 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:19.121631 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e0dcb0ab-rkwgf"] Jul 24 19:11:19.417558 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:19.417521 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2427b70-8f2e-42e8-9235-f80670cc01f8" path="/var/lib/kubelet/pods/e2427b70-8f2e-42e8-9235-f80670cc01f8/volumes" Jul 24 19:11:22.939253 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:22.939217 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc"] Jul 24 19:11:22.939872 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:22.939852 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2427b70-8f2e-42e8-9235-f80670cc01f8" containerName="revoke-keys" Jul 24 19:11:22.939872 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:22.939873 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2427b70-8f2e-42e8-9235-f80670cc01f8" containerName="revoke-keys" Jul 24 19:11:22.940023 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:22.939989 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2427b70-8f2e-42e8-9235-f80670cc01f8" containerName="revoke-keys" Jul 24 19:11:22.943270 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:22.943249 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:22.945983 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:22.945963 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ef3668b1\"" Jul 24 19:11:22.946099 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:22.945963 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ef3668b1-dockercfg-k87lb\"" Jul 24 19:11:22.955361 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:22.955333 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc"] Jul 24 19:11:23.091081 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:23.091038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4w57\" (UniqueName: \"kubernetes.io/projected/ebc80bd5-92d6-4f76-a558-ba267753306c-kube-api-access-x4w57\") pod \"payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc\" (UID: \"ebc80bd5-92d6-4f76-a558-ba267753306c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:23.091081 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:23.091081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc80bd5-92d6-4f76-a558-ba267753306c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc\" (UID: \"ebc80bd5-92d6-4f76-a558-ba267753306c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:23.192271 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:23.192175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4w57\" (UniqueName: \"kubernetes.io/projected/ebc80bd5-92d6-4f76-a558-ba267753306c-kube-api-access-x4w57\") pod \"payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc\" (UID: \"ebc80bd5-92d6-4f76-a558-ba267753306c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:23.192271 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:23.192219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc80bd5-92d6-4f76-a558-ba267753306c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc\" (UID: \"ebc80bd5-92d6-4f76-a558-ba267753306c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:23.193006 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:23.192985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc80bd5-92d6-4f76-a558-ba267753306c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc\" (UID: \"ebc80bd5-92d6-4f76-a558-ba267753306c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:23.201415 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:23.201383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4w57\" (UniqueName: \"kubernetes.io/projected/ebc80bd5-92d6-4f76-a558-ba267753306c-kube-api-access-x4w57\") pod \"payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc\" (UID: \"ebc80bd5-92d6-4f76-a558-ba267753306c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:23.254908 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:23.254873 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:23.414615 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:11:23.414589 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc80bd5_92d6_4f76_a558_ba267753306c.slice/crio-3ce5d2d60872514daa75c712a01b9968241ec1023273316a29e0e25801212465 WatchSource:0}: Error finding container 3ce5d2d60872514daa75c712a01b9968241ec1023273316a29e0e25801212465: Status 404 returned error can't find the container with id 3ce5d2d60872514daa75c712a01b9968241ec1023273316a29e0e25801212465 Jul 24 19:11:23.419195 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:23.419174 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc"] Jul 24 19:11:24.172989 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:24.172953 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" event={"ID":"ebc80bd5-92d6-4f76-a558-ba267753306c","Type":"ContainerStarted","Data":"1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f"} Jul 24 19:11:24.173389 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:24.173008 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" event={"ID":"ebc80bd5-92d6-4f76-a558-ba267753306c","Type":"ContainerStarted","Data":"3ce5d2d60872514daa75c712a01b9968241ec1023273316a29e0e25801212465"} Jul 24 19:11:24.173389 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:24.173035 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:24.188162 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:24.188104 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" podStartSLOduration=2.188087302 podStartE2EDuration="2.188087302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:11:24.186989198 +0000 UTC m=+1679.263501758" watchObservedRunningTime="2026-07-24 19:11:24.188087302 +0000 UTC m=+1679.264599859" Jul 24 19:11:35.181394 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:35.181364 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:47.937647 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:47.937567 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5"] Jul 24 19:11:47.941209 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:47.941192 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5" Jul 24 19:11:47.943680 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:47.943653 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:11:47.948434 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:47.948397 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5"] Jul 24 19:11:48.021185 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:48.021154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba10602-4c8a-48fd-ab1a-39640f861faf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5\" (UID: \"7ba10602-4c8a-48fd-ab1a-39640f861faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5" Jul 24 19:11:48.122165 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:48.122124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba10602-4c8a-48fd-ab1a-39640f861faf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5\" (UID: \"7ba10602-4c8a-48fd-ab1a-39640f861faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5" Jul 24 19:11:48.122759 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:48.122720 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba10602-4c8a-48fd-ab1a-39640f861faf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5\" (UID: \"7ba10602-4c8a-48fd-ab1a-39640f861faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5" Jul 24 19:11:48.254217 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:48.254132 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5" Jul 24 19:11:48.607643 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:48.607560 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5"] Jul 24 19:11:48.610846 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:11:48.610808 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba10602_4c8a_48fd_ab1a_39640f861faf.slice/crio-2764dc130f3e0ced5cbeccdf88cfaece9315775941524316d423c40a0a0eaccb WatchSource:0}: Error finding container 2764dc130f3e0ced5cbeccdf88cfaece9315775941524316d423c40a0a0eaccb: Status 404 returned error can't find the container with id 2764dc130f3e0ced5cbeccdf88cfaece9315775941524316d423c40a0a0eaccb Jul 24 19:11:49.318429 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:49.318399 2575 generic.go:356] "Generic (PLEG): container finished" podID="7ba10602-4c8a-48fd-ab1a-39640f861faf" containerID="ae1ed8a62f0371071504372669d107a160cda462e5284fe5d387c37ffd1aac31" exitCode=0 Jul 24 19:11:49.318774 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:49.318446 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5" event={"ID":"7ba10602-4c8a-48fd-ab1a-39640f861faf","Type":"ContainerDied","Data":"ae1ed8a62f0371071504372669d107a160cda462e5284fe5d387c37ffd1aac31"} Jul 24 19:11:49.318774 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:49.318492 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5" event={"ID":"7ba10602-4c8a-48fd-ab1a-39640f861faf","Type":"ContainerStarted","Data":"2764dc130f3e0ced5cbeccdf88cfaece9315775941524316d423c40a0a0eaccb"} Jul 24 19:11:50.468507 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:50.468482 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5" Jul 24 19:11:50.542955 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:50.542929 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba10602-4c8a-48fd-ab1a-39640f861faf-maas-api-service-ca\") pod \"7ba10602-4c8a-48fd-ab1a-39640f861faf\" (UID: \"7ba10602-4c8a-48fd-ab1a-39640f861faf\") " Jul 24 19:11:50.543263 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:50.543241 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba10602-4c8a-48fd-ab1a-39640f861faf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ba10602-4c8a-48fd-ab1a-39640f861faf" (UID: "7ba10602-4c8a-48fd-ab1a-39640f861faf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:11:50.644102 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:50.644069 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba10602-4c8a-48fd-ab1a-39640f861faf-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:11:51.330359 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:51.330312 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5" event={"ID":"7ba10602-4c8a-48fd-ab1a-39640f861faf","Type":"ContainerDied","Data":"2764dc130f3e0ced5cbeccdf88cfaece9315775941524316d423c40a0a0eaccb"} Jul 24 19:11:51.330359 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:51.330335 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5" Jul 24 19:11:51.330359 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:51.330367 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2764dc130f3e0ced5cbeccdf88cfaece9315775941524316d423c40a0a0eaccb" Jul 24 19:11:58.605231 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:58.605197 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc"] Jul 24 19:11:58.605806 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:58.605449 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" podUID="ebc80bd5-92d6-4f76-a558-ba267753306c" containerName="payload-pre-processing" containerID="cri-o://1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f" gracePeriod=30 Jul 24 19:11:58.861716 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:58.861647 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:58.916593 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:58.916565 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4w57\" (UniqueName: \"kubernetes.io/projected/ebc80bd5-92d6-4f76-a558-ba267753306c-kube-api-access-x4w57\") pod \"ebc80bd5-92d6-4f76-a558-ba267753306c\" (UID: \"ebc80bd5-92d6-4f76-a558-ba267753306c\") " Jul 24 19:11:58.916593 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:58.916599 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc80bd5-92d6-4f76-a558-ba267753306c-plugins-config-volume\") pod \"ebc80bd5-92d6-4f76-a558-ba267753306c\" (UID: \"ebc80bd5-92d6-4f76-a558-ba267753306c\") " Jul 24 19:11:58.917013 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:58.916991 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc80bd5-92d6-4f76-a558-ba267753306c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ebc80bd5-92d6-4f76-a558-ba267753306c" (UID: "ebc80bd5-92d6-4f76-a558-ba267753306c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:11:58.918876 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:58.918851 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc80bd5-92d6-4f76-a558-ba267753306c-kube-api-access-x4w57" (OuterVolumeSpecName: "kube-api-access-x4w57") pod "ebc80bd5-92d6-4f76-a558-ba267753306c" (UID: "ebc80bd5-92d6-4f76-a558-ba267753306c"). InnerVolumeSpecName "kube-api-access-x4w57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:11:59.018233 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.018201 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4w57\" (UniqueName: \"kubernetes.io/projected/ebc80bd5-92d6-4f76-a558-ba267753306c-kube-api-access-x4w57\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:11:59.018233 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.018228 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc80bd5-92d6-4f76-a558-ba267753306c-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:11:59.373455 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.373418 2575 generic.go:356] "Generic (PLEG): container finished" podID="ebc80bd5-92d6-4f76-a558-ba267753306c" containerID="1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f" exitCode=0 Jul 24 19:11:59.373646 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.373464 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" Jul 24 19:11:59.373646 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.373505 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" event={"ID":"ebc80bd5-92d6-4f76-a558-ba267753306c","Type":"ContainerDied","Data":"1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f"} Jul 24 19:11:59.373646 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.373559 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc" event={"ID":"ebc80bd5-92d6-4f76-a558-ba267753306c","Type":"ContainerDied","Data":"3ce5d2d60872514daa75c712a01b9968241ec1023273316a29e0e25801212465"} Jul 24 19:11:59.373646 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.373585 2575 scope.go:117] "RemoveContainer" containerID="1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f" Jul 24 19:11:59.383626 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.383609 2575 scope.go:117] "RemoveContainer" containerID="1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f" Jul 24 19:11:59.383893 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:11:59.383876 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f\": container with ID starting with 1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f not found: ID does not exist" containerID="1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f" Jul 24 19:11:59.383933 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.383902 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f"} err="failed to get container status \"1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f\": rpc error: code = NotFound desc = could not find container \"1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f\": container with ID starting with 1d67d5272b5a3530026508e5908107e1118b3f7e512825d65538ac15ad02ca3f not found: ID does not exist" Jul 24 19:11:59.398012 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.397992 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc"] Jul 24 19:11:59.402389 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.402369 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ef3668b1-76847dd7d4-snfvc"] Jul 24 19:11:59.418003 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:11:59.417973 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc80bd5-92d6-4f76-a558-ba267753306c" path="/var/lib/kubelet/pods/ebc80bd5-92d6-4f76-a558-ba267753306c/volumes" Jul 24 19:12:03.033977 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:03.033944 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5"] Jul 24 19:12:03.047123 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:03.047084 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ef3668b1-dm2z5"] Jul 24 19:12:03.417912 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:03.417861 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba10602-4c8a-48fd-ab1a-39640f861faf" path="/var/lib/kubelet/pods/7ba10602-4c8a-48fd-ab1a-39640f861faf/volumes" Jul 24 19:12:08.610830 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.610800 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p"] Jul 24 19:12:08.611231 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.611209 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba10602-4c8a-48fd-ab1a-39640f861faf" containerName="revoke-keys" Jul 24 19:12:08.611231 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.611220 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba10602-4c8a-48fd-ab1a-39640f861faf" containerName="revoke-keys" Jul 24 19:12:08.611309 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.611241 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc80bd5-92d6-4f76-a558-ba267753306c" containerName="payload-pre-processing" Jul 24 19:12:08.611309 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.611246 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc80bd5-92d6-4f76-a558-ba267753306c" containerName="payload-pre-processing" Jul 24 19:12:08.611375 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.611311 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba10602-4c8a-48fd-ab1a-39640f861faf" containerName="revoke-keys" Jul 24 19:12:08.611375 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.611318 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc80bd5-92d6-4f76-a558-ba267753306c" containerName="payload-pre-processing" Jul 24 19:12:08.615724 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.615708 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:08.618461 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.618440 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-73fbe69e-dockercfg-9bf2b\"" Jul 24 19:12:08.618558 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.618479 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-73fbe69e\"" Jul 24 19:12:08.637480 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.637453 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p"] Jul 24 19:12:08.708198 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.708149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82db8481-7a49-4e5b-b414-ffc85d9e5558-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p\" (UID: \"82db8481-7a49-4e5b-b414-ffc85d9e5558\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:08.708198 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.708200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzfn\" (UniqueName: \"kubernetes.io/projected/82db8481-7a49-4e5b-b414-ffc85d9e5558-kube-api-access-bhzfn\") pod \"payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p\" (UID: \"82db8481-7a49-4e5b-b414-ffc85d9e5558\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:08.809004 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.808961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82db8481-7a49-4e5b-b414-ffc85d9e5558-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p\" (UID: \"82db8481-7a49-4e5b-b414-ffc85d9e5558\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:08.809186 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.809030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzfn\" (UniqueName: \"kubernetes.io/projected/82db8481-7a49-4e5b-b414-ffc85d9e5558-kube-api-access-bhzfn\") pod \"payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p\" (UID: \"82db8481-7a49-4e5b-b414-ffc85d9e5558\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:08.809766 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.809722 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82db8481-7a49-4e5b-b414-ffc85d9e5558-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p\" (UID: \"82db8481-7a49-4e5b-b414-ffc85d9e5558\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:08.822627 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.822595 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzfn\" (UniqueName: \"kubernetes.io/projected/82db8481-7a49-4e5b-b414-ffc85d9e5558-kube-api-access-bhzfn\") pod \"payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p\" (UID: \"82db8481-7a49-4e5b-b414-ffc85d9e5558\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:08.928915 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:08.928886 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:09.306458 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:09.306433 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p"] Jul 24 19:12:09.308350 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:12:09.308320 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82db8481_7a49_4e5b_b414_ffc85d9e5558.slice/crio-24d5fffa54356014d9755514fe1056a9ef7c9c1de0ae3ddae3dd0a556027d111 WatchSource:0}: Error finding container 24d5fffa54356014d9755514fe1056a9ef7c9c1de0ae3ddae3dd0a556027d111: Status 404 returned error can't find the container with id 24d5fffa54356014d9755514fe1056a9ef7c9c1de0ae3ddae3dd0a556027d111 Jul 24 19:12:09.430426 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:09.430367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" event={"ID":"82db8481-7a49-4e5b-b414-ffc85d9e5558","Type":"ContainerStarted","Data":"5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926"} Jul 24 19:12:09.430426 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:09.430426 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" event={"ID":"82db8481-7a49-4e5b-b414-ffc85d9e5558","Type":"ContainerStarted","Data":"24d5fffa54356014d9755514fe1056a9ef7c9c1de0ae3ddae3dd0a556027d111"} Jul 24 19:12:09.430663 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:09.430480 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:09.446887 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:09.446834 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" podStartSLOduration=1.44681817 podStartE2EDuration="1.44681817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:12:09.445194612 +0000 UTC m=+1724.521707170" watchObservedRunningTime="2026-07-24 19:12:09.44681817 +0000 UTC m=+1724.523330728" Jul 24 19:12:20.438938 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:20.438908 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:32.569248 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:32.569215 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm"] Jul 24 19:12:32.574103 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:32.574085 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" Jul 24 19:12:32.577433 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:32.577404 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:12:32.579696 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:32.579601 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm"] Jul 24 19:12:32.628967 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:32.628934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb89740-35e6-4fb2-86ed-3d501292142f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm\" (UID: \"bbb89740-35e6-4fb2-86ed-3d501292142f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" Jul 24 19:12:32.730070 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:32.730035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb89740-35e6-4fb2-86ed-3d501292142f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm\" (UID: \"bbb89740-35e6-4fb2-86ed-3d501292142f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" Jul 24 19:12:32.730648 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:32.730627 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb89740-35e6-4fb2-86ed-3d501292142f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm\" (UID: \"bbb89740-35e6-4fb2-86ed-3d501292142f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" Jul 24 19:12:32.887796 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:32.887769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" Jul 24 19:12:33.038717 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:33.038687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm"] Jul 24 19:12:33.041054 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:12:33.041028 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb89740_35e6_4fb2_86ed_3d501292142f.slice/crio-650ae8919e8d4da84d02a1f1b18d6511e5f0f239010345dc0f57c358c1b452a9 WatchSource:0}: Error finding container 650ae8919e8d4da84d02a1f1b18d6511e5f0f239010345dc0f57c358c1b452a9: Status 404 returned error can't find the container with id 650ae8919e8d4da84d02a1f1b18d6511e5f0f239010345dc0f57c358c1b452a9 Jul 24 19:12:33.564365 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:33.564270 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" event={"ID":"bbb89740-35e6-4fb2-86ed-3d501292142f","Type":"ContainerStarted","Data":"3ec4162f7b212ef39b12d4c9e35c3b092e0a1669189f23c9c5e39c10c47c8f11"} Jul 24 19:12:33.564365 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:33.564320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" event={"ID":"bbb89740-35e6-4fb2-86ed-3d501292142f","Type":"ContainerStarted","Data":"650ae8919e8d4da84d02a1f1b18d6511e5f0f239010345dc0f57c358c1b452a9"} Jul 24 19:12:33.585254 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:33.585066 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" podStartSLOduration=1.585046868 podStartE2EDuration="1.585046868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:12:33.584809512 +0000 UTC m=+1748.661322072" watchObservedRunningTime="2026-07-24 19:12:33.585046868 +0000 UTC m=+1748.661559427" Jul 24 19:12:34.570844 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:34.570812 2575 generic.go:356] "Generic (PLEG): container finished" podID="bbb89740-35e6-4fb2-86ed-3d501292142f" containerID="3ec4162f7b212ef39b12d4c9e35c3b092e0a1669189f23c9c5e39c10c47c8f11" exitCode=0 Jul 24 19:12:34.571019 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:34.570897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" event={"ID":"bbb89740-35e6-4fb2-86ed-3d501292142f","Type":"ContainerDied","Data":"3ec4162f7b212ef39b12d4c9e35c3b092e0a1669189f23c9c5e39c10c47c8f11"} Jul 24 19:12:35.719672 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:35.719644 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" Jul 24 19:12:35.757384 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:35.757359 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb89740-35e6-4fb2-86ed-3d501292142f-maas-api-service-ca\") pod \"bbb89740-35e6-4fb2-86ed-3d501292142f\" (UID: \"bbb89740-35e6-4fb2-86ed-3d501292142f\") " Jul 24 19:12:35.757840 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:35.757794 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb89740-35e6-4fb2-86ed-3d501292142f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bbb89740-35e6-4fb2-86ed-3d501292142f" (UID: "bbb89740-35e6-4fb2-86ed-3d501292142f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:12:35.858081 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:35.858002 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb89740-35e6-4fb2-86ed-3d501292142f-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:12:36.584030 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:36.584005 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" Jul 24 19:12:36.584207 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:36.584003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm" event={"ID":"bbb89740-35e6-4fb2-86ed-3d501292142f","Type":"ContainerDied","Data":"650ae8919e8d4da84d02a1f1b18d6511e5f0f239010345dc0f57c358c1b452a9"} Jul 24 19:12:36.584207 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:36.584114 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="650ae8919e8d4da84d02a1f1b18d6511e5f0f239010345dc0f57c358c1b452a9" Jul 24 19:12:43.174348 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.174311 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p"] Jul 24 19:12:43.174800 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.174602 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" podUID="82db8481-7a49-4e5b-b414-ffc85d9e5558" containerName="payload-pre-processing" containerID="cri-o://5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926" gracePeriod=30 Jul 24 19:12:43.444755 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.444716 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:43.529057 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.529032 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82db8481-7a49-4e5b-b414-ffc85d9e5558-plugins-config-volume\") pod \"82db8481-7a49-4e5b-b414-ffc85d9e5558\" (UID: \"82db8481-7a49-4e5b-b414-ffc85d9e5558\") " Jul 24 19:12:43.529057 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.529063 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhzfn\" (UniqueName: \"kubernetes.io/projected/82db8481-7a49-4e5b-b414-ffc85d9e5558-kube-api-access-bhzfn\") pod \"82db8481-7a49-4e5b-b414-ffc85d9e5558\" (UID: \"82db8481-7a49-4e5b-b414-ffc85d9e5558\") " Jul 24 19:12:43.529501 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.529471 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82db8481-7a49-4e5b-b414-ffc85d9e5558-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82db8481-7a49-4e5b-b414-ffc85d9e5558" (UID: "82db8481-7a49-4e5b-b414-ffc85d9e5558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:12:43.531356 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.531332 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82db8481-7a49-4e5b-b414-ffc85d9e5558-kube-api-access-bhzfn" (OuterVolumeSpecName: "kube-api-access-bhzfn") pod "82db8481-7a49-4e5b-b414-ffc85d9e5558" (UID: "82db8481-7a49-4e5b-b414-ffc85d9e5558"). InnerVolumeSpecName "kube-api-access-bhzfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:12:43.622308 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.622276 2575 generic.go:356] "Generic (PLEG): container finished" podID="82db8481-7a49-4e5b-b414-ffc85d9e5558" containerID="5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926" exitCode=0 Jul 24 19:12:43.622471 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.622324 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" Jul 24 19:12:43.622471 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.622358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" event={"ID":"82db8481-7a49-4e5b-b414-ffc85d9e5558","Type":"ContainerDied","Data":"5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926"} Jul 24 19:12:43.622471 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.622412 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p" event={"ID":"82db8481-7a49-4e5b-b414-ffc85d9e5558","Type":"ContainerDied","Data":"24d5fffa54356014d9755514fe1056a9ef7c9c1de0ae3ddae3dd0a556027d111"} Jul 24 19:12:43.622471 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.622436 2575 scope.go:117] "RemoveContainer" containerID="5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926" Jul 24 19:12:43.630047 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.629855 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82db8481-7a49-4e5b-b414-ffc85d9e5558-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:12:43.630047 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.629873 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhzfn\" (UniqueName: \"kubernetes.io/projected/82db8481-7a49-4e5b-b414-ffc85d9e5558-kube-api-access-bhzfn\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:12:43.631870 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.631856 2575 scope.go:117] "RemoveContainer" containerID="5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926" Jul 24 19:12:43.632135 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:12:43.632114 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926\": container with ID starting with 5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926 not found: ID does not exist" containerID="5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926" Jul 24 19:12:43.632220 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.632142 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926"} err="failed to get container status \"5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926\": rpc error: code = NotFound desc = could not find container \"5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926\": container with ID starting with 5214bc0cfa5db54a1390bec2eb27e0bc87c0ca3522443e25a9197eb172301926 not found: ID does not exist" Jul 24 19:12:43.644126 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.644102 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p"] Jul 24 19:12:43.647065 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:43.647046 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-73fbe69e-7fbc67476-x254p"] Jul 24 19:12:45.417231 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:45.417192 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82db8481-7a49-4e5b-b414-ffc85d9e5558" path="/var/lib/kubelet/pods/82db8481-7a49-4e5b-b414-ffc85d9e5558/volumes" Jul 24 19:12:47.660137 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:47.660087 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm"] Jul 24 19:12:47.662303 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:47.662279 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-73fbe69e-vt2fm"] Jul 24 19:12:49.418800 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:12:49.418764 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbb89740-35e6-4fb2-86ed-3d501292142f" path="/var/lib/kubelet/pods/bbb89740-35e6-4fb2-86ed-3d501292142f/volumes" Jul 24 19:13:25.531618 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:13:25.531591 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:13:25.533185 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:13:25.533150 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:13:25.540304 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:13:25.540282 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:13:25.541694 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:13:25.541678 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:14:51.369782 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.369725 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm"] Jul 24 19:14:51.370426 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.370406 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbb89740-35e6-4fb2-86ed-3d501292142f" containerName="revoke-keys" Jul 24 19:14:51.370472 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.370431 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbb89740-35e6-4fb2-86ed-3d501292142f" containerName="revoke-keys" Jul 24 19:14:51.370472 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.370452 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82db8481-7a49-4e5b-b414-ffc85d9e5558" containerName="payload-pre-processing" Jul 24 19:14:51.370472 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.370461 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="82db8481-7a49-4e5b-b414-ffc85d9e5558" containerName="payload-pre-processing" Jul 24 19:14:51.370578 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.370567 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="82db8481-7a49-4e5b-b414-ffc85d9e5558" containerName="payload-pre-processing" Jul 24 19:14:51.370618 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.370583 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbb89740-35e6-4fb2-86ed-3d501292142f" containerName="revoke-keys" Jul 24 19:14:51.374389 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.374369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" Jul 24 19:14:51.377616 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.377593 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:14:51.380148 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.380126 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm"] Jul 24 19:14:51.484007 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.483969 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b4b08a-be6a-419f-a762-954ad9b1866f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm\" (UID: \"40b4b08a-be6a-419f-a762-954ad9b1866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" Jul 24 19:14:51.585418 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.585386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b4b08a-be6a-419f-a762-954ad9b1866f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm\" (UID: \"40b4b08a-be6a-419f-a762-954ad9b1866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" Jul 24 19:14:51.586033 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.586012 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b4b08a-be6a-419f-a762-954ad9b1866f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm\" (UID: \"40b4b08a-be6a-419f-a762-954ad9b1866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" Jul 24 19:14:51.688562 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.688533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" Jul 24 19:14:51.833124 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:51.833098 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm"] Jul 24 19:14:51.835261 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:14:51.835215 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b4b08a_be6a_419f_a762_954ad9b1866f.slice/crio-b06e6ad76b900c1dc6d37e6417d56adbc6fe67ef5471d9cb3c3e8082041ee060 WatchSource:0}: Error finding container b06e6ad76b900c1dc6d37e6417d56adbc6fe67ef5471d9cb3c3e8082041ee060: Status 404 returned error can't find the container with id b06e6ad76b900c1dc6d37e6417d56adbc6fe67ef5471d9cb3c3e8082041ee060 Jul 24 19:14:52.344656 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:52.344565 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" event={"ID":"40b4b08a-be6a-419f-a762-954ad9b1866f","Type":"ContainerStarted","Data":"62e54c2dcc55f7bdb4eb5e770f5f4ff96192c09b3b9eeb366a0c9537b1caaa33"} Jul 24 19:14:52.344656 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:52.344632 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" event={"ID":"40b4b08a-be6a-419f-a762-954ad9b1866f","Type":"ContainerStarted","Data":"b06e6ad76b900c1dc6d37e6417d56adbc6fe67ef5471d9cb3c3e8082041ee060"} Jul 24 19:14:52.360519 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:52.360474 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" podStartSLOduration=1.360460416 podStartE2EDuration="1.360460416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:14:52.358541944 +0000 UTC m=+1887.435054514" watchObservedRunningTime="2026-07-24 19:14:52.360460416 +0000 UTC m=+1887.436972975" Jul 24 19:14:53.350903 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:53.350863 2575 generic.go:356] "Generic (PLEG): container finished" podID="40b4b08a-be6a-419f-a762-954ad9b1866f" containerID="62e54c2dcc55f7bdb4eb5e770f5f4ff96192c09b3b9eeb366a0c9537b1caaa33" exitCode=0 Jul 24 19:14:53.351326 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:53.350952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" event={"ID":"40b4b08a-be6a-419f-a762-954ad9b1866f","Type":"ContainerDied","Data":"62e54c2dcc55f7bdb4eb5e770f5f4ff96192c09b3b9eeb366a0c9537b1caaa33"} Jul 24 19:14:54.501434 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:54.501411 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" Jul 24 19:14:54.611475 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:54.611401 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b4b08a-be6a-419f-a762-954ad9b1866f-maas-api-service-ca\") pod \"40b4b08a-be6a-419f-a762-954ad9b1866f\" (UID: \"40b4b08a-be6a-419f-a762-954ad9b1866f\") " Jul 24 19:14:54.611732 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:54.611711 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b4b08a-be6a-419f-a762-954ad9b1866f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "40b4b08a-be6a-419f-a762-954ad9b1866f" (UID: "40b4b08a-be6a-419f-a762-954ad9b1866f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:14:54.712753 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:54.712717 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b4b08a-be6a-419f-a762-954ad9b1866f-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:14:55.363666 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:55.363636 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" Jul 24 19:14:55.363861 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:55.363672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm" event={"ID":"40b4b08a-be6a-419f-a762-954ad9b1866f","Type":"ContainerDied","Data":"b06e6ad76b900c1dc6d37e6417d56adbc6fe67ef5471d9cb3c3e8082041ee060"} Jul 24 19:14:55.363861 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:14:55.363718 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b06e6ad76b900c1dc6d37e6417d56adbc6fe67ef5471d9cb3c3e8082041ee060" Jul 24 19:15:00.134726 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.134690 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748675-6q7jk"] Jul 24 19:15:00.135217 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.135184 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b4b08a-be6a-419f-a762-954ad9b1866f" containerName="revoke-keys" Jul 24 19:15:00.135217 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.135199 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b4b08a-be6a-419f-a762-954ad9b1866f" containerName="revoke-keys" Jul 24 19:15:00.135338 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.135266 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="40b4b08a-be6a-419f-a762-954ad9b1866f" containerName="revoke-keys" Jul 24 19:15:00.138368 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.138348 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d"] Jul 24 19:15:00.138503 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.138482 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748675-6q7jk" Jul 24 19:15:00.141181 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.141163 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:15:00.141547 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.141533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d" Jul 24 19:15:00.145902 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.145873 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d"] Jul 24 19:15:00.150369 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.150340 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748675-6q7jk"] Jul 24 19:15:00.152460 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.152441 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748675-6q7jk" Jul 24 19:15:00.160048 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.160019 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d" Jul 24 19:15:00.308415 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.308383 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748675-6q7jk"] Jul 24 19:15:00.335446 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.335383 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d"] Jul 24 19:15:00.336659 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:15:00.336629 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf252eada_8095_4f50_b70c_d33e361b66cf.slice/crio-ac271f362da9f06d44a70b5b2112f71185943491152db30f0508e0f5d9e95299 WatchSource:0}: Error finding container ac271f362da9f06d44a70b5b2112f71185943491152db30f0508e0f5d9e95299: Status 404 returned error can't find the container with id ac271f362da9f06d44a70b5b2112f71185943491152db30f0508e0f5d9e95299 Jul 24 19:15:00.392007 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.391976 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d" event={"ID":"f252eada-8095-4f50-b70c-d33e361b66cf","Type":"ContainerStarted","Data":"ac271f362da9f06d44a70b5b2112f71185943491152db30f0508e0f5d9e95299"} Jul 24 19:15:00.393637 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:00.393607 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748675-6q7jk" event={"ID":"13f45e1d-9790-475e-99af-f25dc3d9bab6","Type":"ContainerStarted","Data":"9eef179822c79c885e5eaf4079772be020ca9a067835a69ea83328b115000aa7"} Jul 24 19:15:01.400635 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:01.400608 2575 generic.go:356] "Generic (PLEG): container finished" podID="f252eada-8095-4f50-b70c-d33e361b66cf" containerID="42fd946b2869d58828a1491e434daa7d4484eb890da803c80b0dca3fc17a1385" exitCode=0 Jul 24 19:15:01.401020 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:01.400695 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d" event={"ID":"f252eada-8095-4f50-b70c-d33e361b66cf","Type":"ContainerDied","Data":"42fd946b2869d58828a1491e434daa7d4484eb890da803c80b0dca3fc17a1385"} Jul 24 19:15:01.402291 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:01.402266 2575 generic.go:356] "Generic (PLEG): container finished" podID="13f45e1d-9790-475e-99af-f25dc3d9bab6" containerID="d20feb9c8dfd07418dacfe7ecc5117e042bfc9579fe666d9699b3e1531e4f82e" exitCode=0 Jul 24 19:15:01.402409 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:01.402325 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748675-6q7jk" event={"ID":"13f45e1d-9790-475e-99af-f25dc3d9bab6","Type":"ContainerDied","Data":"d20feb9c8dfd07418dacfe7ecc5117e042bfc9579fe666d9699b3e1531e4f82e"} Jul 24 19:15:01.699426 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:01.699391 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d"] Jul 24 19:15:02.576852 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:02.576825 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d" Jul 24 19:15:02.580097 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:02.580078 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748675-6q7jk" Jul 24 19:15:03.417389 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:03.417356 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d" Jul 24 19:15:03.418855 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:03.418829 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748675-6q7jk" Jul 24 19:15:03.419863 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:03.419832 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d" event={"ID":"f252eada-8095-4f50-b70c-d33e361b66cf","Type":"ContainerDied","Data":"ac271f362da9f06d44a70b5b2112f71185943491152db30f0508e0f5d9e95299"} Jul 24 19:15:03.419986 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:03.419882 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac271f362da9f06d44a70b5b2112f71185943491152db30f0508e0f5d9e95299" Jul 24 19:15:03.419986 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:03.419897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748675-6q7jk" event={"ID":"13f45e1d-9790-475e-99af-f25dc3d9bab6","Type":"ContainerDied","Data":"9eef179822c79c885e5eaf4079772be020ca9a067835a69ea83328b115000aa7"} Jul 24 19:15:03.419986 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:03.419919 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eef179822c79c885e5eaf4079772be020ca9a067835a69ea83328b115000aa7" Jul 24 19:15:03.444871 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:03.444844 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d"] Jul 24 19:15:03.446720 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:03.446701 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-16b0ae71-29748675-xkt2d"] Jul 24 19:15:05.417412 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:05.417380 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f252eada-8095-4f50-b70c-d33e361b66cf" path="/var/lib/kubelet/pods/f252eada-8095-4f50-b70c-d33e361b66cf/volumes" Jul 24 19:15:06.444289 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:06.444259 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm"] Jul 24 19:15:06.446917 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:06.446885 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-16b0ae71-gx9gm"] Jul 24 19:15:07.419096 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:15:07.419058 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40b4b08a-be6a-419f-a762-954ad9b1866f" path="/var/lib/kubelet/pods/40b4b08a-be6a-419f-a762-954ad9b1866f/volumes" Jul 24 19:16:14.863313 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.863265 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks"] Jul 24 19:16:14.865943 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.863718 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f45e1d-9790-475e-99af-f25dc3d9bab6" containerName="cleanup" Jul 24 19:16:14.865943 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.863732 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f45e1d-9790-475e-99af-f25dc3d9bab6" containerName="cleanup" Jul 24 19:16:14.865943 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.863758 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f252eada-8095-4f50-b70c-d33e361b66cf" containerName="cleanup" Jul 24 19:16:14.865943 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.863763 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f252eada-8095-4f50-b70c-d33e361b66cf" containerName="cleanup" Jul 24 19:16:14.865943 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.863829 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f45e1d-9790-475e-99af-f25dc3d9bab6" containerName="cleanup" Jul 24 19:16:14.865943 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.863837 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f252eada-8095-4f50-b70c-d33e361b66cf" containerName="cleanup" Jul 24 19:16:14.866813 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.866796 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" Jul 24 19:16:14.869554 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.869533 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:16:14.874208 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.874182 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks"] Jul 24 19:16:14.932086 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:14.932050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54c34cb-3672-43d7-8acf-4d5affe3ad6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7baf010d-rsxks\" (UID: \"d54c34cb-3672-43d7-8acf-4d5affe3ad6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" Jul 24 19:16:15.033062 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:15.033027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54c34cb-3672-43d7-8acf-4d5affe3ad6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7baf010d-rsxks\" (UID: \"d54c34cb-3672-43d7-8acf-4d5affe3ad6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" Jul 24 19:16:15.033663 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:15.033640 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54c34cb-3672-43d7-8acf-4d5affe3ad6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7baf010d-rsxks\" (UID: \"d54c34cb-3672-43d7-8acf-4d5affe3ad6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" Jul 24 19:16:15.180590 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:15.180550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" Jul 24 19:16:15.330557 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:15.330530 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks"] Jul 24 19:16:15.333202 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:16:15.333170 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54c34cb_3672_43d7_8acf_4d5affe3ad6d.slice/crio-4a92e73a6d083eec83416bfd689988586eac0c43fc936f0a541657d4603e0123 WatchSource:0}: Error finding container 4a92e73a6d083eec83416bfd689988586eac0c43fc936f0a541657d4603e0123: Status 404 returned error can't find the container with id 4a92e73a6d083eec83416bfd689988586eac0c43fc936f0a541657d4603e0123 Jul 24 19:16:15.814813 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:15.814710 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" event={"ID":"d54c34cb-3672-43d7-8acf-4d5affe3ad6d","Type":"ContainerStarted","Data":"cc8dd5266915b3bef9faa5cb752f7350bcf2b7f8a1e798cfc2a898d52f89a527"} Jul 24 19:16:15.814813 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:15.814801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" event={"ID":"d54c34cb-3672-43d7-8acf-4d5affe3ad6d","Type":"ContainerStarted","Data":"4a92e73a6d083eec83416bfd689988586eac0c43fc936f0a541657d4603e0123"} Jul 24 19:16:15.831704 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:15.831650 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" podStartSLOduration=1.831632554 podStartE2EDuration="1.831632554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:16:15.830043075 +0000 UTC m=+1970.906555633" watchObservedRunningTime="2026-07-24 19:16:15.831632554 +0000 UTC m=+1970.908145111" Jul 24 19:16:16.822570 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:16.822487 2575 generic.go:356] "Generic (PLEG): container finished" podID="d54c34cb-3672-43d7-8acf-4d5affe3ad6d" containerID="cc8dd5266915b3bef9faa5cb752f7350bcf2b7f8a1e798cfc2a898d52f89a527" exitCode=0 Jul 24 19:16:16.822970 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:16.822570 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" event={"ID":"d54c34cb-3672-43d7-8acf-4d5affe3ad6d","Type":"ContainerDied","Data":"cc8dd5266915b3bef9faa5cb752f7350bcf2b7f8a1e798cfc2a898d52f89a527"} Jul 24 19:16:17.968667 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:17.968638 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" Jul 24 19:16:18.060965 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:18.060933 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54c34cb-3672-43d7-8acf-4d5affe3ad6d-maas-api-service-ca\") pod \"d54c34cb-3672-43d7-8acf-4d5affe3ad6d\" (UID: \"d54c34cb-3672-43d7-8acf-4d5affe3ad6d\") " Jul 24 19:16:18.061675 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:18.061639 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54c34cb-3672-43d7-8acf-4d5affe3ad6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d54c34cb-3672-43d7-8acf-4d5affe3ad6d" (UID: "d54c34cb-3672-43d7-8acf-4d5affe3ad6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:16:18.163187 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:18.163154 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d54c34cb-3672-43d7-8acf-4d5affe3ad6d-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:16:18.834769 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:18.834711 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" Jul 24 19:16:18.834953 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:18.834774 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks" event={"ID":"d54c34cb-3672-43d7-8acf-4d5affe3ad6d","Type":"ContainerDied","Data":"4a92e73a6d083eec83416bfd689988586eac0c43fc936f0a541657d4603e0123"} Jul 24 19:16:18.834953 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:18.834828 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a92e73a6d083eec83416bfd689988586eac0c43fc936f0a541657d4603e0123" Jul 24 19:16:25.639957 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:25.639925 2575 scope.go:117] "RemoveContainer" containerID="0e35a63d3d58db722863e9d8c8315d2c4bdf5f8a09f7ea551b03e17444da9ab6" Jul 24 19:16:34.947668 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:34.947633 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks"] Jul 24 19:16:34.950720 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:34.950689 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7baf010d-rsxks"] Jul 24 19:16:35.417592 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:16:35.417561 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d54c34cb-3672-43d7-8acf-4d5affe3ad6d" path="/var/lib/kubelet/pods/d54c34cb-3672-43d7-8acf-4d5affe3ad6d/volumes" Jul 24 19:17:25.660692 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:17:25.660657 2575 scope.go:117] "RemoveContainer" containerID="14110f780cd87f4b3c897794c2bd05c55be37b85feff1da9ac462faed8bbcd40" Jul 24 19:18:25.571165 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:25.571132 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:18:25.574910 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:25.574890 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:18:25.579550 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:25.579530 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:18:25.582882 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:25.582867 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:18:25.682717 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:25.682683 2575 scope.go:117] "RemoveContainer" containerID="ae1ed8a62f0371071504372669d107a160cda462e5284fe5d387c37ffd1aac31" Jul 24 19:18:32.859200 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.859161 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk"] Jul 24 19:18:32.859706 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.859647 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d54c34cb-3672-43d7-8acf-4d5affe3ad6d" containerName="revoke-keys" Jul 24 19:18:32.859706 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.859660 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d54c34cb-3672-43d7-8acf-4d5affe3ad6d" containerName="revoke-keys" Jul 24 19:18:32.859850 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.859734 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d54c34cb-3672-43d7-8acf-4d5affe3ad6d" containerName="revoke-keys" Jul 24 19:18:32.863145 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.863129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:18:32.865718 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.865687 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-948d66-dockercfg-nfsnp\"" Jul 24 19:18:32.865857 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.865757 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-948d66\"" Jul 24 19:18:32.874843 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.874817 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk"] Jul 24 19:18:32.877277 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.877253 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8"] Jul 24 19:18:32.880768 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.880724 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:18:32.898141 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.898115 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8"] Jul 24 19:18:32.970659 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.970628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwtq\" (UniqueName: \"kubernetes.io/projected/567ccd2c-3323-4c2e-9aec-1560927480a1-kube-api-access-6kwtq\") pod \"payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8\" (UID: \"567ccd2c-3323-4c2e-9aec-1560927480a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:18:32.970837 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.970671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dea21f9-7557-49d6-8262-de8a4041db03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk\" (UID: \"2dea21f9-7557-49d6-8262-de8a4041db03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:18:32.970837 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.970781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567ccd2c-3323-4c2e-9aec-1560927480a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8\" (UID: \"567ccd2c-3323-4c2e-9aec-1560927480a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:18:32.970929 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:32.970917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvc98\" (UniqueName: \"kubernetes.io/projected/2dea21f9-7557-49d6-8262-de8a4041db03-kube-api-access-lvc98\") pod \"payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk\" (UID: \"2dea21f9-7557-49d6-8262-de8a4041db03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:18:33.071972 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.071931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567ccd2c-3323-4c2e-9aec-1560927480a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8\" (UID: \"567ccd2c-3323-4c2e-9aec-1560927480a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:18:33.072157 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.072033 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvc98\" (UniqueName: \"kubernetes.io/projected/2dea21f9-7557-49d6-8262-de8a4041db03-kube-api-access-lvc98\") pod \"payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk\" (UID: \"2dea21f9-7557-49d6-8262-de8a4041db03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:18:33.072157 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.072073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwtq\" (UniqueName: \"kubernetes.io/projected/567ccd2c-3323-4c2e-9aec-1560927480a1-kube-api-access-6kwtq\") pod \"payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8\" (UID: \"567ccd2c-3323-4c2e-9aec-1560927480a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:18:33.072157 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.072096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dea21f9-7557-49d6-8262-de8a4041db03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk\" (UID: \"2dea21f9-7557-49d6-8262-de8a4041db03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:18:33.072665 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.072643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567ccd2c-3323-4c2e-9aec-1560927480a1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8\" (UID: \"567ccd2c-3323-4c2e-9aec-1560927480a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:18:33.072817 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.072800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dea21f9-7557-49d6-8262-de8a4041db03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk\" (UID: \"2dea21f9-7557-49d6-8262-de8a4041db03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:18:33.081181 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.081149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwtq\" (UniqueName: \"kubernetes.io/projected/567ccd2c-3323-4c2e-9aec-1560927480a1-kube-api-access-6kwtq\") pod \"payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8\" (UID: \"567ccd2c-3323-4c2e-9aec-1560927480a1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:18:33.081305 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.081189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvc98\" (UniqueName: \"kubernetes.io/projected/2dea21f9-7557-49d6-8262-de8a4041db03-kube-api-access-lvc98\") pod \"payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk\" (UID: \"2dea21f9-7557-49d6-8262-de8a4041db03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:18:33.175371 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.175334 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:18:33.190669 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.190641 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:18:33.344157 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.344009 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk"] Jul 24 19:18:33.344403 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:18:33.344374 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dea21f9_7557_49d6_8262_de8a4041db03.slice/crio-6107171a98f365be600195389361d3783fa3fcdd2c8b9827bdc37b94f5bd2423 WatchSource:0}: Error finding container 6107171a98f365be600195389361d3783fa3fcdd2c8b9827bdc37b94f5bd2423: Status 404 returned error can't find the container with id 6107171a98f365be600195389361d3783fa3fcdd2c8b9827bdc37b94f5bd2423 Jul 24 19:18:33.370723 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.370695 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8"] Jul 24 19:18:33.375047 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:18:33.375023 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567ccd2c_3323_4c2e_9aec_1560927480a1.slice/crio-2243c8a49699e6ac9ce4eab3974e324bff38a180218ca577edd60998c42841cc WatchSource:0}: Error finding container 2243c8a49699e6ac9ce4eab3974e324bff38a180218ca577edd60998c42841cc: Status 404 returned error can't find the container with id 2243c8a49699e6ac9ce4eab3974e324bff38a180218ca577edd60998c42841cc Jul 24 19:18:33.592482 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.592442 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" event={"ID":"2dea21f9-7557-49d6-8262-de8a4041db03","Type":"ContainerStarted","Data":"6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda"} Jul 24 19:18:33.592760 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.592718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" event={"ID":"2dea21f9-7557-49d6-8262-de8a4041db03","Type":"ContainerStarted","Data":"6107171a98f365be600195389361d3783fa3fcdd2c8b9827bdc37b94f5bd2423"} Jul 24 19:18:33.592922 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.592909 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:18:33.595176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.595094 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" event={"ID":"567ccd2c-3323-4c2e-9aec-1560927480a1","Type":"ContainerStarted","Data":"ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e"} Jul 24 19:18:33.595176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.595144 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" event={"ID":"567ccd2c-3323-4c2e-9aec-1560927480a1","Type":"ContainerStarted","Data":"2243c8a49699e6ac9ce4eab3974e324bff38a180218ca577edd60998c42841cc"} Jul 24 19:18:33.595358 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.595191 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:18:33.611010 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.610954 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" podStartSLOduration=1.610937572 podStartE2EDuration="1.610937572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:18:33.608671042 +0000 UTC m=+2108.685183604" watchObservedRunningTime="2026-07-24 19:18:33.610937572 +0000 UTC m=+2108.687450138" Jul 24 19:18:33.627803 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:33.627721 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" podStartSLOduration=1.6277064430000001 podStartE2EDuration="1.627706443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:18:33.625993805 +0000 UTC m=+2108.702506367" watchObservedRunningTime="2026-07-24 19:18:33.627706443 +0000 UTC m=+2108.704219057" Jul 24 19:18:44.603602 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:44.603558 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:18:45.609910 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:18:45.609878 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:19:25.705686 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:19:25.705609 2575 scope.go:117] "RemoveContainer" containerID="3ec4162f7b212ef39b12d4c9e35c3b092e0a1669189f23c9c5e39c10c47c8f11" Jul 24 19:20:02.514704 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:02.514664 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5"] Jul 24 19:20:02.518549 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:02.518527 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5" Jul 24 19:20:02.521956 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:02.521928 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:20:02.529016 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:02.528536 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5"] Jul 24 19:20:02.594910 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:02.594873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfc43e5-e275-486a-b1a5-b8a2b08e82c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5\" (UID: \"6dfc43e5-e275-486a-b1a5-b8a2b08e82c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5" Jul 24 19:20:02.695940 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:02.695900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfc43e5-e275-486a-b1a5-b8a2b08e82c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5\" (UID: \"6dfc43e5-e275-486a-b1a5-b8a2b08e82c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5" Jul 24 19:20:02.696491 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:02.696471 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfc43e5-e275-486a-b1a5-b8a2b08e82c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5\" (UID: \"6dfc43e5-e275-486a-b1a5-b8a2b08e82c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5" Jul 24 19:20:02.832509 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:02.832432 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5" Jul 24 19:20:02.982328 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:02.982298 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5"] Jul 24 19:20:02.983721 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:20:02.983697 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dfc43e5_e275_486a_b1a5_b8a2b08e82c0.slice/crio-676e9c0c0bfeb20e13e3409ac23a84f1dee5df96902896c7db4b071cfff1e75d WatchSource:0}: Error finding container 676e9c0c0bfeb20e13e3409ac23a84f1dee5df96902896c7db4b071cfff1e75d: Status 404 returned error can't find the container with id 676e9c0c0bfeb20e13e3409ac23a84f1dee5df96902896c7db4b071cfff1e75d Jul 24 19:20:03.109333 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:03.109242 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5" event={"ID":"6dfc43e5-e275-486a-b1a5-b8a2b08e82c0","Type":"ContainerStarted","Data":"676e9c0c0bfeb20e13e3409ac23a84f1dee5df96902896c7db4b071cfff1e75d"} Jul 24 19:20:04.116579 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:04.116494 2575 generic.go:356] "Generic (PLEG): container finished" podID="6dfc43e5-e275-486a-b1a5-b8a2b08e82c0" containerID="cf4d9bd9f137aa9891dcdbddf295faf04ff2079f560be0956dfbbf2e69c18725" exitCode=0 Jul 24 19:20:04.116579 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:04.116569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5" event={"ID":"6dfc43e5-e275-486a-b1a5-b8a2b08e82c0","Type":"ContainerDied","Data":"cf4d9bd9f137aa9891dcdbddf295faf04ff2079f560be0956dfbbf2e69c18725"} Jul 24 19:20:05.262349 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:05.262323 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5" Jul 24 19:20:05.320058 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:05.320023 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfc43e5-e275-486a-b1a5-b8a2b08e82c0-maas-api-service-ca\") pod \"6dfc43e5-e275-486a-b1a5-b8a2b08e82c0\" (UID: \"6dfc43e5-e275-486a-b1a5-b8a2b08e82c0\") " Jul 24 19:20:05.320388 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:05.320366 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dfc43e5-e275-486a-b1a5-b8a2b08e82c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6dfc43e5-e275-486a-b1a5-b8a2b08e82c0" (UID: "6dfc43e5-e275-486a-b1a5-b8a2b08e82c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:20:05.421452 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:05.421422 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dfc43e5-e275-486a-b1a5-b8a2b08e82c0-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:20:06.136443 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:06.136409 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5" Jul 24 19:20:06.136634 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:06.136441 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5" event={"ID":"6dfc43e5-e275-486a-b1a5-b8a2b08e82c0","Type":"ContainerDied","Data":"676e9c0c0bfeb20e13e3409ac23a84f1dee5df96902896c7db4b071cfff1e75d"} Jul 24 19:20:06.136634 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:06.136488 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="676e9c0c0bfeb20e13e3409ac23a84f1dee5df96902896c7db4b071cfff1e75d" Jul 24 19:20:17.614992 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:17.614946 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5"] Jul 24 19:20:17.619774 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:17.619731 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-950cf0-9zrv5"] Jul 24 19:20:19.418181 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:19.418147 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dfc43e5-e275-486a-b1a5-b8a2b08e82c0" path="/var/lib/kubelet/pods/6dfc43e5-e275-486a-b1a5-b8a2b08e82c0/volumes" Jul 24 19:20:34.996893 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:34.996856 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8"] Jul 24 19:20:34.997480 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:34.997144 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" podUID="567ccd2c-3323-4c2e-9aec-1560927480a1" containerName="payload-processing" containerID="cri-o://ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e" gracePeriod=30 Jul 24 19:20:35.005122 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.005093 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk"] Jul 24 19:20:35.005501 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.005474 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" podUID="2dea21f9-7557-49d6-8262-de8a4041db03" containerName="payload-pre-processing" containerID="cri-o://6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda" gracePeriod=30 Jul 24 19:20:35.290883 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.290856 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:20:35.293836 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.293804 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:20:35.308228 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.308196 2575 generic.go:356] "Generic (PLEG): container finished" podID="2dea21f9-7557-49d6-8262-de8a4041db03" containerID="6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda" exitCode=0 Jul 24 19:20:35.308415 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.308260 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" Jul 24 19:20:35.308415 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.308339 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" event={"ID":"2dea21f9-7557-49d6-8262-de8a4041db03","Type":"ContainerDied","Data":"6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda"} Jul 24 19:20:35.308540 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.308419 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk" event={"ID":"2dea21f9-7557-49d6-8262-de8a4041db03","Type":"ContainerDied","Data":"6107171a98f365be600195389361d3783fa3fcdd2c8b9827bdc37b94f5bd2423"} Jul 24 19:20:35.308540 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.308449 2575 scope.go:117] "RemoveContainer" containerID="6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda" Jul 24 19:20:35.311093 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.311070 2575 generic.go:356] "Generic (PLEG): container finished" podID="567ccd2c-3323-4c2e-9aec-1560927480a1" containerID="ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e" exitCode=0 Jul 24 19:20:35.311234 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.311188 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" Jul 24 19:20:35.311406 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.311203 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" event={"ID":"567ccd2c-3323-4c2e-9aec-1560927480a1","Type":"ContainerDied","Data":"ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e"} Jul 24 19:20:35.311485 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.311438 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8" event={"ID":"567ccd2c-3323-4c2e-9aec-1560927480a1","Type":"ContainerDied","Data":"2243c8a49699e6ac9ce4eab3974e324bff38a180218ca577edd60998c42841cc"} Jul 24 19:20:35.319165 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.319143 2575 scope.go:117] "RemoveContainer" containerID="6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda" Jul 24 19:20:35.319445 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:20:35.319422 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda\": container with ID starting with 6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda not found: ID does not exist" containerID="6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda" Jul 24 19:20:35.319524 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.319454 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda"} err="failed to get container status \"6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda\": rpc error: code = NotFound desc = could not find container \"6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda\": container with ID starting with 6fb5dc8aa5f2b57051b71284bdd264efb7e969c40af0b8d460a170d20af05eda not found: ID does not exist" Jul 24 19:20:35.319524 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.319471 2575 scope.go:117] "RemoveContainer" containerID="ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e" Jul 24 19:20:35.330101 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.330077 2575 scope.go:117] "RemoveContainer" containerID="ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e" Jul 24 19:20:35.330455 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:20:35.330428 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e\": container with ID starting with ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e not found: ID does not exist" containerID="ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e" Jul 24 19:20:35.330544 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.330462 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e"} err="failed to get container status \"ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e\": rpc error: code = NotFound desc = could not find container \"ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e\": container with ID starting with ca755f51e2c831bf7829012a0e59b926296e7f9560f8e4a9298b2df18788564e not found: ID does not exist" Jul 24 19:20:35.405391 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.405357 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kwtq\" (UniqueName: \"kubernetes.io/projected/567ccd2c-3323-4c2e-9aec-1560927480a1-kube-api-access-6kwtq\") pod \"567ccd2c-3323-4c2e-9aec-1560927480a1\" (UID: \"567ccd2c-3323-4c2e-9aec-1560927480a1\") " Jul 24 19:20:35.405594 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.405436 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567ccd2c-3323-4c2e-9aec-1560927480a1-plugins-config-volume\") pod \"567ccd2c-3323-4c2e-9aec-1560927480a1\" (UID: \"567ccd2c-3323-4c2e-9aec-1560927480a1\") " Jul 24 19:20:35.405594 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.405469 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dea21f9-7557-49d6-8262-de8a4041db03-plugins-config-volume\") pod \"2dea21f9-7557-49d6-8262-de8a4041db03\" (UID: \"2dea21f9-7557-49d6-8262-de8a4041db03\") " Jul 24 19:20:35.405594 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.405519 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvc98\" (UniqueName: \"kubernetes.io/projected/2dea21f9-7557-49d6-8262-de8a4041db03-kube-api-access-lvc98\") pod \"2dea21f9-7557-49d6-8262-de8a4041db03\" (UID: \"2dea21f9-7557-49d6-8262-de8a4041db03\") " Jul 24 19:20:35.405884 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.405853 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dea21f9-7557-49d6-8262-de8a4041db03-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2dea21f9-7557-49d6-8262-de8a4041db03" (UID: "2dea21f9-7557-49d6-8262-de8a4041db03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:20:35.405998 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.405887 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567ccd2c-3323-4c2e-9aec-1560927480a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "567ccd2c-3323-4c2e-9aec-1560927480a1" (UID: "567ccd2c-3323-4c2e-9aec-1560927480a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:20:35.407700 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.407678 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567ccd2c-3323-4c2e-9aec-1560927480a1-kube-api-access-6kwtq" (OuterVolumeSpecName: "kube-api-access-6kwtq") pod "567ccd2c-3323-4c2e-9aec-1560927480a1" (UID: "567ccd2c-3323-4c2e-9aec-1560927480a1"). InnerVolumeSpecName "kube-api-access-6kwtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:20:35.407890 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.407869 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dea21f9-7557-49d6-8262-de8a4041db03-kube-api-access-lvc98" (OuterVolumeSpecName: "kube-api-access-lvc98") pod "2dea21f9-7557-49d6-8262-de8a4041db03" (UID: "2dea21f9-7557-49d6-8262-de8a4041db03"). InnerVolumeSpecName "kube-api-access-lvc98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:20:35.507133 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.507084 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwtq\" (UniqueName: \"kubernetes.io/projected/567ccd2c-3323-4c2e-9aec-1560927480a1-kube-api-access-6kwtq\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:20:35.507133 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.507133 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/567ccd2c-3323-4c2e-9aec-1560927480a1-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:20:35.507133 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.507144 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dea21f9-7557-49d6-8262-de8a4041db03-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:20:35.507133 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.507156 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvc98\" (UniqueName: \"kubernetes.io/projected/2dea21f9-7557-49d6-8262-de8a4041db03-kube-api-access-lvc98\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:20:35.623989 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.623908 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk"] Jul 24 19:20:35.628964 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.628936 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-948d66-9886d89f6-bn4rk"] Jul 24 19:20:35.639711 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.639681 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8"] Jul 24 19:20:35.645729 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:35.645704 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-948d66-d8fb5d694-wlrf8"] Jul 24 19:20:37.417339 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:37.417301 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dea21f9-7557-49d6-8262-de8a4041db03" path="/var/lib/kubelet/pods/2dea21f9-7557-49d6-8262-de8a4041db03/volumes" Jul 24 19:20:37.417866 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:20:37.417847 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="567ccd2c-3323-4c2e-9aec-1560927480a1" path="/var/lib/kubelet/pods/567ccd2c-3323-4c2e-9aec-1560927480a1/volumes" Jul 24 19:21:00.005311 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.005269 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm"] Jul 24 19:21:00.005901 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.005882 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dfc43e5-e275-486a-b1a5-b8a2b08e82c0" containerName="revoke-keys" Jul 24 19:21:00.006004 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.005904 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dfc43e5-e275-486a-b1a5-b8a2b08e82c0" containerName="revoke-keys" Jul 24 19:21:00.006004 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.005938 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567ccd2c-3323-4c2e-9aec-1560927480a1" containerName="payload-processing" Jul 24 19:21:00.006004 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.005947 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="567ccd2c-3323-4c2e-9aec-1560927480a1" containerName="payload-processing" Jul 24 19:21:00.006004 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.005975 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dea21f9-7557-49d6-8262-de8a4041db03" containerName="payload-pre-processing" Jul 24 19:21:00.006004 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.005984 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dea21f9-7557-49d6-8262-de8a4041db03" containerName="payload-pre-processing" Jul 24 19:21:00.006244 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.006090 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dfc43e5-e275-486a-b1a5-b8a2b08e82c0" containerName="revoke-keys" Jul 24 19:21:00.006244 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.006107 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dea21f9-7557-49d6-8262-de8a4041db03" containerName="payload-pre-processing" Jul 24 19:21:00.006244 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.006118 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="567ccd2c-3323-4c2e-9aec-1560927480a1" containerName="payload-processing" Jul 24 19:21:00.009306 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.009285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:00.012293 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.012266 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ff6bd9\"" Jul 24 19:21:00.012397 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.012344 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ff6bd9-dockercfg-mcqt5\"" Jul 24 19:21:00.028645 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.028615 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7x5f\" (UniqueName: \"kubernetes.io/projected/8d04e2f4-5c8c-4690-bf93-b6fba642f752-kube-api-access-f7x5f\") pod \"payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm\" (UID: \"8d04e2f4-5c8c-4690-bf93-b6fba642f752\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:00.028938 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.028916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04e2f4-5c8c-4690-bf93-b6fba642f752-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm\" (UID: \"8d04e2f4-5c8c-4690-bf93-b6fba642f752\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:00.029958 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.029936 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm"] Jul 24 19:21:00.129502 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.129465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7x5f\" (UniqueName: \"kubernetes.io/projected/8d04e2f4-5c8c-4690-bf93-b6fba642f752-kube-api-access-f7x5f\") pod \"payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm\" (UID: \"8d04e2f4-5c8c-4690-bf93-b6fba642f752\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:00.129689 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.129599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04e2f4-5c8c-4690-bf93-b6fba642f752-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm\" (UID: \"8d04e2f4-5c8c-4690-bf93-b6fba642f752\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:00.130297 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.130275 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04e2f4-5c8c-4690-bf93-b6fba642f752-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm\" (UID: \"8d04e2f4-5c8c-4690-bf93-b6fba642f752\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:00.141406 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.141373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7x5f\" (UniqueName: \"kubernetes.io/projected/8d04e2f4-5c8c-4690-bf93-b6fba642f752-kube-api-access-f7x5f\") pod \"payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm\" (UID: \"8d04e2f4-5c8c-4690-bf93-b6fba642f752\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:00.320112 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.320022 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:00.474493 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:00.474469 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm"] Jul 24 19:21:00.476575 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:21:00.476542 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d04e2f4_5c8c_4690_bf93_b6fba642f752.slice/crio-1218d13027a89ae11d2efde77e18f0b8acd70bf1156b9da3fab4c6eb0c805cb4 WatchSource:0}: Error finding container 1218d13027a89ae11d2efde77e18f0b8acd70bf1156b9da3fab4c6eb0c805cb4: Status 404 returned error can't find the container with id 1218d13027a89ae11d2efde77e18f0b8acd70bf1156b9da3fab4c6eb0c805cb4 Jul 24 19:21:01.460801 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:01.460763 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" event={"ID":"8d04e2f4-5c8c-4690-bf93-b6fba642f752","Type":"ContainerStarted","Data":"03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf"} Jul 24 19:21:01.461210 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:01.460824 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" event={"ID":"8d04e2f4-5c8c-4690-bf93-b6fba642f752","Type":"ContainerStarted","Data":"1218d13027a89ae11d2efde77e18f0b8acd70bf1156b9da3fab4c6eb0c805cb4"} Jul 24 19:21:01.461210 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:01.460873 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:01.477024 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:01.476974 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" podStartSLOduration=2.476957015 podStartE2EDuration="2.476957015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:21:01.475675053 +0000 UTC m=+2256.552187612" watchObservedRunningTime="2026-07-24 19:21:01.476957015 +0000 UTC m=+2256.553469572" Jul 24 19:21:11.386419 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:11.386383 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8"] Jul 24 19:21:11.390581 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:11.390559 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" Jul 24 19:21:11.393094 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:11.393068 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:21:11.402828 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:11.399176 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8"] Jul 24 19:21:11.429559 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:11.429511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c86ac-79fa-4523-a7b6-7a571d835752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8\" (UID: \"2d4c86ac-79fa-4523-a7b6-7a571d835752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" Jul 24 19:21:11.530811 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:11.530779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c86ac-79fa-4523-a7b6-7a571d835752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8\" (UID: \"2d4c86ac-79fa-4523-a7b6-7a571d835752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" Jul 24 19:21:11.531431 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:11.531410 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c86ac-79fa-4523-a7b6-7a571d835752-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8\" (UID: \"2d4c86ac-79fa-4523-a7b6-7a571d835752\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" Jul 24 19:21:11.706307 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:11.706209 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" Jul 24 19:21:11.853518 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:11.853491 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8"] Jul 24 19:21:12.469243 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:12.469211 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:12.523286 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:12.523254 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" event={"ID":"2d4c86ac-79fa-4523-a7b6-7a571d835752","Type":"ContainerStarted","Data":"376fc0b3b9f7b7d57ee16520a69a4eabceaee17e8e95706b4a45e469869cdf56"} Jul 24 19:21:12.523286 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:12.523303 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" event={"ID":"2d4c86ac-79fa-4523-a7b6-7a571d835752","Type":"ContainerStarted","Data":"11fd9c53281299cd6291b4d1610ba132611a0f0b87ad44db50f3200933fbf1bc"} Jul 24 19:21:12.540198 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:12.540136 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" podStartSLOduration=1.5401193260000001 podStartE2EDuration="1.540119326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:21:12.539430248 +0000 UTC m=+2267.615942819" watchObservedRunningTime="2026-07-24 19:21:12.540119326 +0000 UTC m=+2267.616631884" Jul 24 19:21:13.531251 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:13.531213 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d4c86ac-79fa-4523-a7b6-7a571d835752" containerID="376fc0b3b9f7b7d57ee16520a69a4eabceaee17e8e95706b4a45e469869cdf56" exitCode=0 Jul 24 19:21:13.531718 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:13.531300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" event={"ID":"2d4c86ac-79fa-4523-a7b6-7a571d835752","Type":"ContainerDied","Data":"376fc0b3b9f7b7d57ee16520a69a4eabceaee17e8e95706b4a45e469869cdf56"} Jul 24 19:21:14.678129 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:14.678099 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" Jul 24 19:21:14.762903 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:14.762862 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c86ac-79fa-4523-a7b6-7a571d835752-maas-api-service-ca\") pod \"2d4c86ac-79fa-4523-a7b6-7a571d835752\" (UID: \"2d4c86ac-79fa-4523-a7b6-7a571d835752\") " Jul 24 19:21:14.763262 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:14.763229 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4c86ac-79fa-4523-a7b6-7a571d835752-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d4c86ac-79fa-4523-a7b6-7a571d835752" (UID: "2d4c86ac-79fa-4523-a7b6-7a571d835752"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:21:14.863878 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:14.863698 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4c86ac-79fa-4523-a7b6-7a571d835752-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:21:15.543065 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:15.543033 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" Jul 24 19:21:15.543242 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:15.543069 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8" event={"ID":"2d4c86ac-79fa-4523-a7b6-7a571d835752","Type":"ContainerDied","Data":"11fd9c53281299cd6291b4d1610ba132611a0f0b87ad44db50f3200933fbf1bc"} Jul 24 19:21:15.543242 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:15.543120 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11fd9c53281299cd6291b4d1610ba132611a0f0b87ad44db50f3200933fbf1bc" Jul 24 19:21:25.736647 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:25.736609 2575 scope.go:117] "RemoveContainer" containerID="42fd946b2869d58828a1491e434daa7d4484eb890da803c80b0dca3fc17a1385" Jul 24 19:21:25.745564 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:25.745541 2575 scope.go:117] "RemoveContainer" containerID="62e54c2dcc55f7bdb4eb5e770f5f4ff96192c09b3b9eeb366a0c9537b1caaa33" Jul 24 19:21:31.476854 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:31.476813 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8"] Jul 24 19:21:31.479381 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:31.479356 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bd2a2-vcmb8"] Jul 24 19:21:33.423466 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:33.423433 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d4c86ac-79fa-4523-a7b6-7a571d835752" path="/var/lib/kubelet/pods/2d4c86ac-79fa-4523-a7b6-7a571d835752/volumes" Jul 24 19:21:38.302184 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.302147 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb"] Jul 24 19:21:38.302708 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.302618 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4c86ac-79fa-4523-a7b6-7a571d835752" containerName="revoke-keys" Jul 24 19:21:38.302708 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.302630 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4c86ac-79fa-4523-a7b6-7a571d835752" containerName="revoke-keys" Jul 24 19:21:38.302807 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.302729 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d4c86ac-79fa-4523-a7b6-7a571d835752" containerName="revoke-keys" Jul 24 19:21:38.306305 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.306288 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" Jul 24 19:21:38.310198 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.310179 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:21:38.331790 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.331723 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb"] Jul 24 19:21:38.486539 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.486480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb\" (UID: \"96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" Jul 24 19:21:38.587193 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.587103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb\" (UID: \"96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" Jul 24 19:21:38.587733 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.587711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb\" (UID: \"96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" Jul 24 19:21:38.617205 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.617169 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" Jul 24 19:21:38.971292 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:38.971267 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb"] Jul 24 19:21:38.973803 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:21:38.973727 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96aaaf18_9993_4efe_9aa3_c0bbfc3b77b8.slice/crio-1a6be1e2830b6d8b8b111db82710ce50e95b1757eb8fa2e771a86db3b591b3f1 WatchSource:0}: Error finding container 1a6be1e2830b6d8b8b111db82710ce50e95b1757eb8fa2e771a86db3b591b3f1: Status 404 returned error can't find the container with id 1a6be1e2830b6d8b8b111db82710ce50e95b1757eb8fa2e771a86db3b591b3f1 Jul 24 19:21:39.679357 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:39.679315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" event={"ID":"96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8","Type":"ContainerStarted","Data":"98f238766d06fd75926dd95977b2d6c156eda017ba7e057cadce0f61e49b908b"} Jul 24 19:21:39.679832 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:39.679383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" event={"ID":"96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8","Type":"ContainerStarted","Data":"1a6be1e2830b6d8b8b111db82710ce50e95b1757eb8fa2e771a86db3b591b3f1"} Jul 24 19:21:39.701529 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:39.701476 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" podStartSLOduration=1.701458661 podStartE2EDuration="1.701458661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:21:39.699278821 +0000 UTC m=+2294.775791410" watchObservedRunningTime="2026-07-24 19:21:39.701458661 +0000 UTC m=+2294.777971218" Jul 24 19:21:40.686214 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:40.686173 2575 generic.go:356] "Generic (PLEG): container finished" podID="96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8" containerID="98f238766d06fd75926dd95977b2d6c156eda017ba7e057cadce0f61e49b908b" exitCode=0 Jul 24 19:21:40.686657 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:40.686266 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" event={"ID":"96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8","Type":"ContainerDied","Data":"98f238766d06fd75926dd95977b2d6c156eda017ba7e057cadce0f61e49b908b"} Jul 24 19:21:41.835043 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:41.835013 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" Jul 24 19:21:42.017621 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:42.017522 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8-maas-api-service-ca\") pod \"96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8\" (UID: \"96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8\") " Jul 24 19:21:42.017934 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:42.017910 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8" (UID: "96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:21:42.118941 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:42.118908 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:21:42.702678 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:42.702643 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" event={"ID":"96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8","Type":"ContainerDied","Data":"1a6be1e2830b6d8b8b111db82710ce50e95b1757eb8fa2e771a86db3b591b3f1"} Jul 24 19:21:42.702678 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:42.702678 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb" Jul 24 19:21:42.702908 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:42.702702 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a6be1e2830b6d8b8b111db82710ce50e95b1757eb8fa2e771a86db3b591b3f1" Jul 24 19:21:53.773121 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:53.773088 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm"] Jul 24 19:21:53.773564 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:53.773340 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" podUID="8d04e2f4-5c8c-4690-bf93-b6fba642f752" containerName="payload-pre-processing" containerID="cri-o://03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf" gracePeriod=30 Jul 24 19:21:54.041553 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.041525 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:54.126848 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.126806 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04e2f4-5c8c-4690-bf93-b6fba642f752-plugins-config-volume\") pod \"8d04e2f4-5c8c-4690-bf93-b6fba642f752\" (UID: \"8d04e2f4-5c8c-4690-bf93-b6fba642f752\") " Jul 24 19:21:54.126848 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.126859 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7x5f\" (UniqueName: \"kubernetes.io/projected/8d04e2f4-5c8c-4690-bf93-b6fba642f752-kube-api-access-f7x5f\") pod \"8d04e2f4-5c8c-4690-bf93-b6fba642f752\" (UID: \"8d04e2f4-5c8c-4690-bf93-b6fba642f752\") " Jul 24 19:21:54.127190 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.127168 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d04e2f4-5c8c-4690-bf93-b6fba642f752-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d04e2f4-5c8c-4690-bf93-b6fba642f752" (UID: "8d04e2f4-5c8c-4690-bf93-b6fba642f752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:21:54.129123 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.129089 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d04e2f4-5c8c-4690-bf93-b6fba642f752-kube-api-access-f7x5f" (OuterVolumeSpecName: "kube-api-access-f7x5f") pod "8d04e2f4-5c8c-4690-bf93-b6fba642f752" (UID: "8d04e2f4-5c8c-4690-bf93-b6fba642f752"). InnerVolumeSpecName "kube-api-access-f7x5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:21:54.227798 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.227759 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d04e2f4-5c8c-4690-bf93-b6fba642f752-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:21:54.227798 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.227797 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7x5f\" (UniqueName: \"kubernetes.io/projected/8d04e2f4-5c8c-4690-bf93-b6fba642f752-kube-api-access-f7x5f\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:21:54.770856 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.770821 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d04e2f4-5c8c-4690-bf93-b6fba642f752" containerID="03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf" exitCode=0 Jul 24 19:21:54.771047 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.770870 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" Jul 24 19:21:54.771047 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.770906 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" event={"ID":"8d04e2f4-5c8c-4690-bf93-b6fba642f752","Type":"ContainerDied","Data":"03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf"} Jul 24 19:21:54.771047 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.770960 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm" event={"ID":"8d04e2f4-5c8c-4690-bf93-b6fba642f752","Type":"ContainerDied","Data":"1218d13027a89ae11d2efde77e18f0b8acd70bf1156b9da3fab4c6eb0c805cb4"} Jul 24 19:21:54.771047 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.770981 2575 scope.go:117] "RemoveContainer" containerID="03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf" Jul 24 19:21:54.781010 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.780733 2575 scope.go:117] "RemoveContainer" containerID="03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf" Jul 24 19:21:54.781334 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:21:54.781027 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf\": container with ID starting with 03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf not found: ID does not exist" containerID="03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf" Jul 24 19:21:54.781334 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.781055 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf"} err="failed to get container status \"03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf\": rpc error: code = NotFound desc = could not find container \"03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf\": container with ID starting with 03acc83aab641ca589aee6ba2b6a6b043fc1af052467a1cddbc464698afe53bf not found: ID does not exist" Jul 24 19:21:54.792890 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.792866 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm"] Jul 24 19:21:54.799132 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:54.799103 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ff6bd9-556bbd7856-r25fm"] Jul 24 19:21:55.417671 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:55.417637 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d04e2f4-5c8c-4690-bf93-b6fba642f752" path="/var/lib/kubelet/pods/8d04e2f4-5c8c-4690-bf93-b6fba642f752/volumes" Jul 24 19:21:58.364378 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:58.364347 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb"] Jul 24 19:21:58.367623 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:58.367589 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ff6bd9-snxfb"] Jul 24 19:21:59.417923 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:21:59.417894 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8" path="/var/lib/kubelet/pods/96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8/volumes" Jul 24 19:22:09.997232 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:09.997191 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw"] Jul 24 19:22:09.997714 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:09.997640 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8" containerName="revoke-keys" Jul 24 19:22:09.997714 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:09.997659 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8" containerName="revoke-keys" Jul 24 19:22:09.997714 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:09.997676 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d04e2f4-5c8c-4690-bf93-b6fba642f752" containerName="payload-pre-processing" Jul 24 19:22:09.997714 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:09.997681 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d04e2f4-5c8c-4690-bf93-b6fba642f752" containerName="payload-pre-processing" Jul 24 19:22:09.997874 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:09.997780 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="96aaaf18-9993-4efe-9aa3-c0bbfc3b77b8" containerName="revoke-keys" Jul 24 19:22:09.997874 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:09.997792 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d04e2f4-5c8c-4690-bf93-b6fba642f752" containerName="payload-pre-processing" Jul 24 19:22:10.001145 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.001129 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:22:10.004033 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.004012 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-e05f4d-dockercfg-rhn57\"" Jul 24 19:22:10.004141 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.004052 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-e05f4d\"" Jul 24 19:22:10.010505 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.010477 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw"] Jul 24 19:22:10.076727 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.076688 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw\" (UID: \"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:22:10.076927 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.076866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxh6j\" (UniqueName: \"kubernetes.io/projected/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-kube-api-access-lxh6j\") pod \"payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw\" (UID: \"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:22:10.177528 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.177490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw\" (UID: \"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:22:10.177718 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.177584 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxh6j\" (UniqueName: \"kubernetes.io/projected/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-kube-api-access-lxh6j\") pod \"payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw\" (UID: \"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:22:10.178313 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.178291 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw\" (UID: \"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:22:10.185857 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.185834 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxh6j\" (UniqueName: \"kubernetes.io/projected/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-kube-api-access-lxh6j\") pod \"payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw\" (UID: \"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:22:10.312239 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.312151 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:22:10.469295 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.469270 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw"] Jul 24 19:22:10.471341 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:22:10.471309 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad20cdd_cc9b_4274_afeb_0e7af63f5ce1.slice/crio-b8da0e58835e950edc43774ea6751a98cd2a34f7a13e9384c447a97735b65732 WatchSource:0}: Error finding container b8da0e58835e950edc43774ea6751a98cd2a34f7a13e9384c447a97735b65732: Status 404 returned error can't find the container with id b8da0e58835e950edc43774ea6751a98cd2a34f7a13e9384c447a97735b65732 Jul 24 19:22:10.860633 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.860596 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" event={"ID":"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1","Type":"ContainerStarted","Data":"aa4ea5599808226503a26b779597332b184657bdfbd88421712a88b8f394a32f"} Jul 24 19:22:10.860831 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.860659 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" event={"ID":"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1","Type":"ContainerStarted","Data":"b8da0e58835e950edc43774ea6751a98cd2a34f7a13e9384c447a97735b65732"} Jul 24 19:22:10.860923 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.860837 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:22:10.876628 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:10.876577 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" podStartSLOduration=1.876561908 podStartE2EDuration="1.876561908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:22:10.875612163 +0000 UTC m=+2325.952124718" watchObservedRunningTime="2026-07-24 19:22:10.876561908 +0000 UTC m=+2325.953074523" Jul 24 19:22:17.981287 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:17.981207 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm"] Jul 24 19:22:17.987832 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:17.986115 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:22:17.990259 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:17.990231 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-48f4d2-dockercfg-ntttx\"" Jul 24 19:22:17.990259 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:17.990240 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-48f4d2\"" Jul 24 19:22:17.998002 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:17.997979 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm"] Jul 24 19:22:18.154354 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.154315 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/383e19d6-bd28-4c53-9061-35e36a903243-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm\" (UID: \"383e19d6-bd28-4c53-9061-35e36a903243\") " pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:22:18.154354 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.154358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5qg\" (UniqueName: \"kubernetes.io/projected/383e19d6-bd28-4c53-9061-35e36a903243-kube-api-access-kf5qg\") pod \"payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm\" (UID: \"383e19d6-bd28-4c53-9061-35e36a903243\") " pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:22:18.255442 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.255355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/383e19d6-bd28-4c53-9061-35e36a903243-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm\" (UID: \"383e19d6-bd28-4c53-9061-35e36a903243\") " pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:22:18.255442 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.255395 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5qg\" (UniqueName: \"kubernetes.io/projected/383e19d6-bd28-4c53-9061-35e36a903243-kube-api-access-kf5qg\") pod \"payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm\" (UID: \"383e19d6-bd28-4c53-9061-35e36a903243\") " pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:22:18.256103 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.256082 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/383e19d6-bd28-4c53-9061-35e36a903243-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm\" (UID: \"383e19d6-bd28-4c53-9061-35e36a903243\") " pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:22:18.263589 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.263566 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5qg\" (UniqueName: \"kubernetes.io/projected/383e19d6-bd28-4c53-9061-35e36a903243-kube-api-access-kf5qg\") pod \"payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm\" (UID: \"383e19d6-bd28-4c53-9061-35e36a903243\") " pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:22:18.303858 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.303832 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:22:18.484835 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.484792 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm"] Jul 24 19:22:18.496446 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:22:18.496417 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383e19d6_bd28_4c53_9061_35e36a903243.slice/crio-6ff5403758d89e908a9d2175c7f59b9f2bd6f55f44bf1af9d32a23c300153e7c WatchSource:0}: Error finding container 6ff5403758d89e908a9d2175c7f59b9f2bd6f55f44bf1af9d32a23c300153e7c: Status 404 returned error can't find the container with id 6ff5403758d89e908a9d2175c7f59b9f2bd6f55f44bf1af9d32a23c300153e7c Jul 24 19:22:18.908274 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.908233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" event={"ID":"383e19d6-bd28-4c53-9061-35e36a903243","Type":"ContainerStarted","Data":"8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79"} Jul 24 19:22:18.908438 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.908293 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" event={"ID":"383e19d6-bd28-4c53-9061-35e36a903243","Type":"ContainerStarted","Data":"6ff5403758d89e908a9d2175c7f59b9f2bd6f55f44bf1af9d32a23c300153e7c"} Jul 24 19:22:18.908438 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.908327 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:22:18.926055 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:18.926006 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" podStartSLOduration=1.92599063 podStartE2EDuration="1.92599063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:22:18.924909596 +0000 UTC m=+2334.001422155" watchObservedRunningTime="2026-07-24 19:22:18.92599063 +0000 UTC m=+2334.002503188" Jul 24 19:22:21.871219 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:21.871184 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:22:25.778162 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:25.778128 2575 scope.go:117] "RemoveContainer" containerID="cc8dd5266915b3bef9faa5cb752f7350bcf2b7f8a1e798cfc2a898d52f89a527" Jul 24 19:22:29.916571 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:22:29.916537 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:23:25.605540 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:23:25.605511 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:23:25.611017 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:23:25.610996 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:23:25.613324 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:23:25.613304 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:23:25.621839 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:23:25.621822 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:26:25.829366 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:26:25.829330 2575 scope.go:117] "RemoveContainer" containerID="cf4d9bd9f137aa9891dcdbddf295faf04ff2079f560be0956dfbbf2e69c18725" Jul 24 19:27:25.851875 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:25.851835 2575 scope.go:117] "RemoveContainer" containerID="376fc0b3b9f7b7d57ee16520a69a4eabceaee17e8e95706b4a45e469869cdf56" Jul 24 19:27:54.535097 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:54.535058 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-139848806885552"] Jul 24 19:27:54.539059 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:54.539035 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806885552" Jul 24 19:27:54.541875 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:54.541839 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5lllf\"" Jul 24 19:27:54.555986 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:54.555921 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-139848806885552"] Jul 24 19:27:54.626218 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:54.626184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxdtm\" (UniqueName: \"kubernetes.io/projected/324fd15d-4386-48a1-ae44-64e5e2faf219-kube-api-access-wxdtm\") pod \"test-curl-7401-139848806885552\" (UID: \"324fd15d-4386-48a1-ae44-64e5e2faf219\") " pod="opendatahub/test-curl-7401-139848806885552" Jul 24 19:27:54.727551 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:54.727519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdtm\" (UniqueName: \"kubernetes.io/projected/324fd15d-4386-48a1-ae44-64e5e2faf219-kube-api-access-wxdtm\") pod \"test-curl-7401-139848806885552\" (UID: \"324fd15d-4386-48a1-ae44-64e5e2faf219\") " pod="opendatahub/test-curl-7401-139848806885552" Jul 24 19:27:54.736019 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:54.735994 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdtm\" (UniqueName: \"kubernetes.io/projected/324fd15d-4386-48a1-ae44-64e5e2faf219-kube-api-access-wxdtm\") pod \"test-curl-7401-139848806885552\" (UID: \"324fd15d-4386-48a1-ae44-64e5e2faf219\") " pod="opendatahub/test-curl-7401-139848806885552" Jul 24 19:27:54.863176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:54.863094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806885552" Jul 24 19:27:55.006556 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:55.006528 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-139848806885552"] Jul 24 19:27:55.008544 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:27:55.008516 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod324fd15d_4386_48a1_ae44_64e5e2faf219.slice/crio-a3881b0b92f6f817ec07e5f8cfda3a60a9845d6c50e07b16c7cecde359ade1fa WatchSource:0}: Error finding container a3881b0b92f6f817ec07e5f8cfda3a60a9845d6c50e07b16c7cecde359ade1fa: Status 404 returned error can't find the container with id a3881b0b92f6f817ec07e5f8cfda3a60a9845d6c50e07b16c7cecde359ade1fa Jul 24 19:27:55.010099 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:55.010078 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 19:27:55.778387 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:55.778350 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806885552" event={"ID":"324fd15d-4386-48a1-ae44-64e5e2faf219","Type":"ContainerStarted","Data":"8a7315f5c18079e3267664fe0abaa9a85acc1b14ce4119b2f4e27e6b2b4d7c9d"} Jul 24 19:27:55.778387 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:55.778391 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806885552" event={"ID":"324fd15d-4386-48a1-ae44-64e5e2faf219","Type":"ContainerStarted","Data":"a3881b0b92f6f817ec07e5f8cfda3a60a9845d6c50e07b16c7cecde359ade1fa"} Jul 24 19:27:55.802565 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:55.802507 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7401-139848806885552" podStartSLOduration=1.122842708 podStartE2EDuration="1.802491786s" totalImagesPullingTime="679.649078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:27:55.801578208 +0000 UTC m=+2670.878090780" watchObservedRunningTime="2026-07-24 19:27:55.802491786 +0000 UTC m=+2670.879004343" Jul 24 19:27:55.936987 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:55.936956 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-139848806885552_324fd15d-4386-48a1-ae44-64e5e2faf219/test-curl-7401-139848806885552/0.log" Jul 24 19:27:56.785665 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:56.785581 2575 generic.go:356] "Generic (PLEG): container finished" podID="324fd15d-4386-48a1-ae44-64e5e2faf219" containerID="8a7315f5c18079e3267664fe0abaa9a85acc1b14ce4119b2f4e27e6b2b4d7c9d" exitCode=0 Jul 24 19:27:56.786077 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:56.785665 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806885552" event={"ID":"324fd15d-4386-48a1-ae44-64e5e2faf219","Type":"ContainerDied","Data":"8a7315f5c18079e3267664fe0abaa9a85acc1b14ce4119b2f4e27e6b2b4d7c9d"} Jul 24 19:27:57.933972 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:57.933948 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806885552" Jul 24 19:27:57.968217 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:57.968183 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-139848806885552"] Jul 24 19:27:57.972640 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:57.972615 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-139848806885552"] Jul 24 19:27:58.057387 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.057280 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxdtm\" (UniqueName: \"kubernetes.io/projected/324fd15d-4386-48a1-ae44-64e5e2faf219-kube-api-access-wxdtm\") pod \"324fd15d-4386-48a1-ae44-64e5e2faf219\" (UID: \"324fd15d-4386-48a1-ae44-64e5e2faf219\") " Jul 24 19:27:58.059480 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.059457 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324fd15d-4386-48a1-ae44-64e5e2faf219-kube-api-access-wxdtm" (OuterVolumeSpecName: "kube-api-access-wxdtm") pod "324fd15d-4386-48a1-ae44-64e5e2faf219" (UID: "324fd15d-4386-48a1-ae44-64e5e2faf219"). InnerVolumeSpecName "kube-api-access-wxdtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:27:58.099474 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.099444 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-139848806912944"] Jul 24 19:27:58.099960 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.099942 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="324fd15d-4386-48a1-ae44-64e5e2faf219" containerName="test-curl-7401-139848806885552" Jul 24 19:27:58.100059 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.099963 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="324fd15d-4386-48a1-ae44-64e5e2faf219" containerName="test-curl-7401-139848806885552" Jul 24 19:27:58.100123 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.100101 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="324fd15d-4386-48a1-ae44-64e5e2faf219" containerName="test-curl-7401-139848806885552" Jul 24 19:27:58.103199 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.103179 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806912944" Jul 24 19:27:58.108590 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.108546 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-139848806912944"] Jul 24 19:27:58.158555 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.158530 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxdtm\" (UniqueName: \"kubernetes.io/projected/324fd15d-4386-48a1-ae44-64e5e2faf219-kube-api-access-wxdtm\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:27:58.259189 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.259156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjhgh\" (UniqueName: \"kubernetes.io/projected/9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9-kube-api-access-sjhgh\") pod \"test-curl-7401-139848806912944\" (UID: \"9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9\") " pod="opendatahub/test-curl-7401-139848806912944" Jul 24 19:27:58.359643 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.359557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjhgh\" (UniqueName: \"kubernetes.io/projected/9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9-kube-api-access-sjhgh\") pod \"test-curl-7401-139848806912944\" (UID: \"9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9\") " pod="opendatahub/test-curl-7401-139848806912944" Jul 24 19:27:58.369160 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.369126 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjhgh\" (UniqueName: \"kubernetes.io/projected/9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9-kube-api-access-sjhgh\") pod \"test-curl-7401-139848806912944\" (UID: \"9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9\") " pod="opendatahub/test-curl-7401-139848806912944" Jul 24 19:27:58.414890 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.414863 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806912944" Jul 24 19:27:58.554459 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.554431 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-139848806912944"] Jul 24 19:27:58.556109 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:27:58.556082 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fce6121_1b1f_4a6d_8f9a_4c37bbf406f9.slice/crio-aea8de06f1c6c7922b58393606e1105d8122b18278377be7de020999cf1c1c07 WatchSource:0}: Error finding container aea8de06f1c6c7922b58393606e1105d8122b18278377be7de020999cf1c1c07: Status 404 returned error can't find the container with id aea8de06f1c6c7922b58393606e1105d8122b18278377be7de020999cf1c1c07 Jul 24 19:27:58.798115 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.798079 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806912944" event={"ID":"9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9","Type":"ContainerStarted","Data":"aea8de06f1c6c7922b58393606e1105d8122b18278377be7de020999cf1c1c07"} Jul 24 19:27:58.799429 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.799406 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806885552" Jul 24 19:27:58.799562 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:58.799455 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3881b0b92f6f817ec07e5f8cfda3a60a9845d6c50e07b16c7cecde359ade1fa" Jul 24 19:27:58.809836 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:27:58.809811 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7401-139848806885552\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="324fd15d-4386-48a1-ae44-64e5e2faf219" pod="opendatahub/test-curl-7401-139848806885552" Jul 24 19:27:59.417950 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:59.417916 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="324fd15d-4386-48a1-ae44-64e5e2faf219" path="/var/lib/kubelet/pods/324fd15d-4386-48a1-ae44-64e5e2faf219/volumes" Jul 24 19:27:59.807801 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:59.807711 2575 generic.go:356] "Generic (PLEG): container finished" podID="9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9" containerID="3d1eda3b942dca89367acfc3fa8a928a93ef921ccd3b224c9cbef95fa04f140c" exitCode=0 Jul 24 19:27:59.807944 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:27:59.807799 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806912944" event={"ID":"9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9","Type":"ContainerDied","Data":"3d1eda3b942dca89367acfc3fa8a928a93ef921ccd3b224c9cbef95fa04f140c"} Jul 24 19:28:00.963422 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:00.963397 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806912944" Jul 24 19:28:00.986005 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:00.985979 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-139848806912944_9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9/test-curl-7401-139848806912944/0.log" Jul 24 19:28:01.011094 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.011065 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-139848806912944"] Jul 24 19:28:01.014642 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.014619 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-139848806912944"] Jul 24 19:28:01.084197 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.084122 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjhgh\" (UniqueName: \"kubernetes.io/projected/9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9-kube-api-access-sjhgh\") pod \"9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9\" (UID: \"9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9\") " Jul 24 19:28:01.086280 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.086255 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9-kube-api-access-sjhgh" (OuterVolumeSpecName: "kube-api-access-sjhgh") pod "9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9" (UID: "9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9"). InnerVolumeSpecName "kube-api-access-sjhgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:28:01.185568 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.185529 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjhgh\" (UniqueName: \"kubernetes.io/projected/9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9-kube-api-access-sjhgh\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:28:01.214230 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.214198 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-139848806995840"] Jul 24 19:28:01.214635 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.214622 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9" containerName="test-curl-7401-139848806912944" Jul 24 19:28:01.214680 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.214637 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9" containerName="test-curl-7401-139848806912944" Jul 24 19:28:01.214734 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.214725 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9" containerName="test-curl-7401-139848806912944" Jul 24 19:28:01.217897 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.217881 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:01.223647 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.223621 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-139848806995840"] Jul 24 19:28:01.387607 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.387569 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxhh2\" (UniqueName: \"kubernetes.io/projected/695e9e21-2486-4cf3-9746-dbf06515fdbe-kube-api-access-sxhh2\") pod \"test-curl-7401-139848806995840\" (UID: \"695e9e21-2486-4cf3-9746-dbf06515fdbe\") " pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:01.418568 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.418524 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9" path="/var/lib/kubelet/pods/9fce6121-1b1f-4a6d-8f9a-4c37bbf406f9/volumes" Jul 24 19:28:01.489092 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.489062 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxhh2\" (UniqueName: \"kubernetes.io/projected/695e9e21-2486-4cf3-9746-dbf06515fdbe-kube-api-access-sxhh2\") pod \"test-curl-7401-139848806995840\" (UID: \"695e9e21-2486-4cf3-9746-dbf06515fdbe\") " pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:01.497536 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.497515 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxhh2\" (UniqueName: \"kubernetes.io/projected/695e9e21-2486-4cf3-9746-dbf06515fdbe-kube-api-access-sxhh2\") pod \"test-curl-7401-139848806995840\" (UID: \"695e9e21-2486-4cf3-9746-dbf06515fdbe\") " pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:01.528452 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.528422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:01.668077 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.668021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-139848806995840"] Jul 24 19:28:01.670390 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:28:01.670367 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod695e9e21_2486_4cf3_9746_dbf06515fdbe.slice/crio-0fcbcdac5a1d59edf59606cbc2edc8ce46e2c4ae69d05913db8fad6819d2818c WatchSource:0}: Error finding container 0fcbcdac5a1d59edf59606cbc2edc8ce46e2c4ae69d05913db8fad6819d2818c: Status 404 returned error can't find the container with id 0fcbcdac5a1d59edf59606cbc2edc8ce46e2c4ae69d05913db8fad6819d2818c Jul 24 19:28:01.818986 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.818959 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806912944" Jul 24 19:28:01.819139 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.819001 2575 scope.go:117] "RemoveContainer" containerID="3d1eda3b942dca89367acfc3fa8a928a93ef921ccd3b224c9cbef95fa04f140c" Jul 24 19:28:01.820293 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:01.820247 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806995840" event={"ID":"695e9e21-2486-4cf3-9746-dbf06515fdbe","Type":"ContainerStarted","Data":"0fcbcdac5a1d59edf59606cbc2edc8ce46e2c4ae69d05913db8fad6819d2818c"} Jul 24 19:28:02.828145 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:02.828107 2575 generic.go:356] "Generic (PLEG): container finished" podID="695e9e21-2486-4cf3-9746-dbf06515fdbe" containerID="071dea72a8c90ef9e081a976b08f0f817a579cac3fc542ee40939c598f7cd0cb" exitCode=0 Jul 24 19:28:02.828513 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:02.828194 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806995840" event={"ID":"695e9e21-2486-4cf3-9746-dbf06515fdbe","Type":"ContainerDied","Data":"071dea72a8c90ef9e081a976b08f0f817a579cac3fc542ee40939c598f7cd0cb"} Jul 24 19:28:04.003671 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.003646 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:04.015527 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.015505 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxhh2\" (UniqueName: \"kubernetes.io/projected/695e9e21-2486-4cf3-9746-dbf06515fdbe-kube-api-access-sxhh2\") pod \"695e9e21-2486-4cf3-9746-dbf06515fdbe\" (UID: \"695e9e21-2486-4cf3-9746-dbf06515fdbe\") " Jul 24 19:28:04.017642 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.017614 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695e9e21-2486-4cf3-9746-dbf06515fdbe-kube-api-access-sxhh2" (OuterVolumeSpecName: "kube-api-access-sxhh2") pod "695e9e21-2486-4cf3-9746-dbf06515fdbe" (UID: "695e9e21-2486-4cf3-9746-dbf06515fdbe"). InnerVolumeSpecName "kube-api-access-sxhh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:28:04.024358 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.024334 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-139848806995840_695e9e21-2486-4cf3-9746-dbf06515fdbe/test-curl-7401-139848806995840/0.log" Jul 24 19:28:04.044584 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.044548 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-139848806995840"] Jul 24 19:28:04.050343 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.050321 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-139848806995840"] Jul 24 19:28:04.117025 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.116949 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxhh2\" (UniqueName: \"kubernetes.io/projected/695e9e21-2486-4cf3-9746-dbf06515fdbe-kube-api-access-sxhh2\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:28:04.173314 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.173283 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-139848806995840"] Jul 24 19:28:04.173730 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.173717 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="695e9e21-2486-4cf3-9746-dbf06515fdbe" containerName="test-curl-7401-139848806995840" Jul 24 19:28:04.173802 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.173732 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="695e9e21-2486-4cf3-9746-dbf06515fdbe" containerName="test-curl-7401-139848806995840" Jul 24 19:28:04.173837 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.173813 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="695e9e21-2486-4cf3-9746-dbf06515fdbe" containerName="test-curl-7401-139848806995840" Jul 24 19:28:04.183437 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.183409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:04.184399 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.184377 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-139848806995840"] Jul 24 19:28:04.217994 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.217952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628pt\" (UniqueName: \"kubernetes.io/projected/934c0f22-0eac-4dff-b5e4-8787ad73da4b-kube-api-access-628pt\") pod \"test-curl-7401-139848806995840\" (UID: \"934c0f22-0eac-4dff-b5e4-8787ad73da4b\") " pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:04.319384 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.319347 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628pt\" (UniqueName: \"kubernetes.io/projected/934c0f22-0eac-4dff-b5e4-8787ad73da4b-kube-api-access-628pt\") pod \"test-curl-7401-139848806995840\" (UID: \"934c0f22-0eac-4dff-b5e4-8787ad73da4b\") " pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:04.327684 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.327651 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-628pt\" (UniqueName: \"kubernetes.io/projected/934c0f22-0eac-4dff-b5e4-8787ad73da4b-kube-api-access-628pt\") pod \"test-curl-7401-139848806995840\" (UID: \"934c0f22-0eac-4dff-b5e4-8787ad73da4b\") " pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:04.493544 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.493516 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:04.639628 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.639603 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-139848806995840"] Jul 24 19:28:04.641329 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:28:04.641300 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod934c0f22_0eac_4dff_b5e4_8787ad73da4b.slice/crio-cd617a478fd0656f15a0b6252dfc8c793a97688bd3aa8d9b1e3be696b7c37ac8 WatchSource:0}: Error finding container cd617a478fd0656f15a0b6252dfc8c793a97688bd3aa8d9b1e3be696b7c37ac8: Status 404 returned error can't find the container with id cd617a478fd0656f15a0b6252dfc8c793a97688bd3aa8d9b1e3be696b7c37ac8 Jul 24 19:28:04.840213 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.840128 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:04.840369 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.840137 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fcbcdac5a1d59edf59606cbc2edc8ce46e2c4ae69d05913db8fad6819d2818c" Jul 24 19:28:04.841384 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.841362 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806995840" event={"ID":"934c0f22-0eac-4dff-b5e4-8787ad73da4b","Type":"ContainerStarted","Data":"cd617a478fd0656f15a0b6252dfc8c793a97688bd3aa8d9b1e3be696b7c37ac8"} Jul 24 19:28:04.852199 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:04.852167 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7401-139848806995840" oldPodUID="695e9e21-2486-4cf3-9746-dbf06515fdbe" podUID="934c0f22-0eac-4dff-b5e4-8787ad73da4b" Jul 24 19:28:05.418910 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:05.418836 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="695e9e21-2486-4cf3-9746-dbf06515fdbe" path="/var/lib/kubelet/pods/695e9e21-2486-4cf3-9746-dbf06515fdbe/volumes" Jul 24 19:28:05.848375 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:05.848293 2575 generic.go:356] "Generic (PLEG): container finished" podID="934c0f22-0eac-4dff-b5e4-8787ad73da4b" containerID="7f5107cdafeefd4b9ada8d4da50008a53239f93fa2722e42a5914c2ae2821cfc" exitCode=0 Jul 24 19:28:05.848523 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:05.848385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806995840" event={"ID":"934c0f22-0eac-4dff-b5e4-8787ad73da4b","Type":"ContainerDied","Data":"7f5107cdafeefd4b9ada8d4da50008a53239f93fa2722e42a5914c2ae2821cfc"} Jul 24 19:28:07.021834 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.021810 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:07.041838 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.041804 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-139848806995840_934c0f22-0eac-4dff-b5e4-8787ad73da4b/test-curl-7401-139848806995840/0.log" Jul 24 19:28:07.043977 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.043956 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-628pt\" (UniqueName: \"kubernetes.io/projected/934c0f22-0eac-4dff-b5e4-8787ad73da4b-kube-api-access-628pt\") pod \"934c0f22-0eac-4dff-b5e4-8787ad73da4b\" (UID: \"934c0f22-0eac-4dff-b5e4-8787ad73da4b\") " Jul 24 19:28:07.046181 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.046153 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934c0f22-0eac-4dff-b5e4-8787ad73da4b-kube-api-access-628pt" (OuterVolumeSpecName: "kube-api-access-628pt") pod "934c0f22-0eac-4dff-b5e4-8787ad73da4b" (UID: "934c0f22-0eac-4dff-b5e4-8787ad73da4b"). InnerVolumeSpecName "kube-api-access-628pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:28:07.078757 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.078715 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-139848806995840"] Jul 24 19:28:07.095729 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.095701 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-139848806995840"] Jul 24 19:28:07.145156 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.145129 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-628pt\" (UniqueName: \"kubernetes.io/projected/934c0f22-0eac-4dff-b5e4-8787ad73da4b-kube-api-access-628pt\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:28:07.224833 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.224792 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7401-139848806996848"] Jul 24 19:28:07.225289 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.225274 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934c0f22-0eac-4dff-b5e4-8787ad73da4b" containerName="test-curl-7401-139848806995840" Jul 24 19:28:07.225359 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.225291 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="934c0f22-0eac-4dff-b5e4-8787ad73da4b" containerName="test-curl-7401-139848806995840" Jul 24 19:28:07.225398 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.225374 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="934c0f22-0eac-4dff-b5e4-8787ad73da4b" containerName="test-curl-7401-139848806995840" Jul 24 19:28:07.228472 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.228457 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806996848" Jul 24 19:28:07.235162 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.235137 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-139848806996848"] Jul 24 19:28:07.245972 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.245945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7c7t\" (UniqueName: \"kubernetes.io/projected/1f1b03d6-f4d3-43a5-8850-7882fd638076-kube-api-access-j7c7t\") pod \"test-curl-7401-139848806996848\" (UID: \"1f1b03d6-f4d3-43a5-8850-7882fd638076\") " pod="opendatahub/test-curl-7401-139848806996848" Jul 24 19:28:07.346915 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.346877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7c7t\" (UniqueName: \"kubernetes.io/projected/1f1b03d6-f4d3-43a5-8850-7882fd638076-kube-api-access-j7c7t\") pod \"test-curl-7401-139848806996848\" (UID: \"1f1b03d6-f4d3-43a5-8850-7882fd638076\") " pod="opendatahub/test-curl-7401-139848806996848" Jul 24 19:28:07.356299 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.356271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7c7t\" (UniqueName: \"kubernetes.io/projected/1f1b03d6-f4d3-43a5-8850-7882fd638076-kube-api-access-j7c7t\") pod \"test-curl-7401-139848806996848\" (UID: \"1f1b03d6-f4d3-43a5-8850-7882fd638076\") " pod="opendatahub/test-curl-7401-139848806996848" Jul 24 19:28:07.418242 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.418170 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934c0f22-0eac-4dff-b5e4-8787ad73da4b" path="/var/lib/kubelet/pods/934c0f22-0eac-4dff-b5e4-8787ad73da4b/volumes" Jul 24 19:28:07.539708 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.539671 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806996848" Jul 24 19:28:07.681785 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.681758 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7401-139848806996848"] Jul 24 19:28:07.683463 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:28:07.683438 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f1b03d6_f4d3_43a5_8850_7882fd638076.slice/crio-866301aef79a9267c2be441fc3e6abdc197590be9e73d07f0518a82afd57cf57 WatchSource:0}: Error finding container 866301aef79a9267c2be441fc3e6abdc197590be9e73d07f0518a82afd57cf57: Status 404 returned error can't find the container with id 866301aef79a9267c2be441fc3e6abdc197590be9e73d07f0518a82afd57cf57 Jul 24 19:28:07.861750 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.861703 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806996848" event={"ID":"1f1b03d6-f4d3-43a5-8850-7882fd638076","Type":"ContainerStarted","Data":"866301aef79a9267c2be441fc3e6abdc197590be9e73d07f0518a82afd57cf57"} Jul 24 19:28:07.862975 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.862956 2575 scope.go:117] "RemoveContainer" containerID="7f5107cdafeefd4b9ada8d4da50008a53239f93fa2722e42a5914c2ae2821cfc" Jul 24 19:28:07.863070 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:07.862981 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806995840" Jul 24 19:28:08.870309 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:08.870276 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f1b03d6-f4d3-43a5-8850-7882fd638076" containerID="2c13b9066971a971be9234bb7ec2f0d394d626272851c4c14237a469cf677c25" exitCode=0 Jul 24 19:28:08.870796 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:08.870368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7401-139848806996848" event={"ID":"1f1b03d6-f4d3-43a5-8850-7882fd638076","Type":"ContainerDied","Data":"2c13b9066971a971be9234bb7ec2f0d394d626272851c4c14237a469cf677c25"} Jul 24 19:28:10.028026 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:10.028004 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806996848" Jul 24 19:28:10.046802 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:10.046772 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7401-139848806996848_1f1b03d6-f4d3-43a5-8850-7882fd638076/test-curl-7401-139848806996848/0.log" Jul 24 19:28:10.067153 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:10.067126 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7401-139848806996848"] Jul 24 19:28:10.070790 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:10.070767 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7401-139848806996848"] Jul 24 19:28:10.072667 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:10.072650 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7c7t\" (UniqueName: \"kubernetes.io/projected/1f1b03d6-f4d3-43a5-8850-7882fd638076-kube-api-access-j7c7t\") pod \"1f1b03d6-f4d3-43a5-8850-7882fd638076\" (UID: \"1f1b03d6-f4d3-43a5-8850-7882fd638076\") " Jul 24 19:28:10.074885 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:10.074859 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1b03d6-f4d3-43a5-8850-7882fd638076-kube-api-access-j7c7t" (OuterVolumeSpecName: "kube-api-access-j7c7t") pod "1f1b03d6-f4d3-43a5-8850-7882fd638076" (UID: "1f1b03d6-f4d3-43a5-8850-7882fd638076"). InnerVolumeSpecName "kube-api-access-j7c7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:28:10.173974 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:10.173943 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7c7t\" (UniqueName: \"kubernetes.io/projected/1f1b03d6-f4d3-43a5-8850-7882fd638076-kube-api-access-j7c7t\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:28:10.883977 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:10.883945 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7401-139848806996848" Jul 24 19:28:10.883977 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:10.883975 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866301aef79a9267c2be441fc3e6abdc197590be9e73d07f0518a82afd57cf57" Jul 24 19:28:10.893847 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:28:10.893818 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7401-139848806996848\" is forbidden: User \"system:node:ip-10-0-134-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-30.ec2.internal' and this object" podUID="1f1b03d6-f4d3-43a5-8850-7882fd638076" pod="opendatahub/test-curl-7401-139848806996848" Jul 24 19:28:11.418768 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:11.418715 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f1b03d6-f4d3-43a5-8850-7882fd638076" path="/var/lib/kubelet/pods/1f1b03d6-f4d3-43a5-8850-7882fd638076/volumes" Jul 24 19:28:25.641036 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:25.640964 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:28:25.649020 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:25.648997 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:28:25.651357 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:25.651337 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:28:25.658897 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:25.658879 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8qzpr_635e0f85-09ee-4cd3-a549-4d5fb3db9b48/ovn-acl-logging/0.log" Jul 24 19:28:25.872600 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:25.872567 2575 scope.go:117] "RemoveContainer" containerID="98f238766d06fd75926dd95977b2d6c156eda017ba7e057cadce0f61e49b908b" Jul 24 19:28:36.909732 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:36.909695 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv"] Jul 24 19:28:36.910171 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:36.910157 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f1b03d6-f4d3-43a5-8850-7882fd638076" containerName="test-curl-7401-139848806996848" Jul 24 19:28:36.910218 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:36.910173 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f1b03d6-f4d3-43a5-8850-7882fd638076" containerName="test-curl-7401-139848806996848" Jul 24 19:28:36.910270 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:36.910260 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f1b03d6-f4d3-43a5-8850-7882fd638076" containerName="test-curl-7401-139848806996848" Jul 24 19:28:36.913531 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:36.913510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:28:36.916249 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:36.916228 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5ce07c-dockercfg-9nr5k\"" Jul 24 19:28:36.916249 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:36.916238 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5ce07c\"" Jul 24 19:28:36.923074 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:36.923049 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv"] Jul 24 19:28:37.025098 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:37.025064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhfrh\" (UniqueName: \"kubernetes.io/projected/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-kube-api-access-nhfrh\") pod \"payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv\" (UID: \"ecf97bb5-80cb-4c33-8fda-080ef6433b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:28:37.025272 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:37.025170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv\" (UID: \"ecf97bb5-80cb-4c33-8fda-080ef6433b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:28:37.125984 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:37.125948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv\" (UID: \"ecf97bb5-80cb-4c33-8fda-080ef6433b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:28:37.126178 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:37.126032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhfrh\" (UniqueName: \"kubernetes.io/projected/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-kube-api-access-nhfrh\") pod \"payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv\" (UID: \"ecf97bb5-80cb-4c33-8fda-080ef6433b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:28:37.126693 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:37.126673 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv\" (UID: \"ecf97bb5-80cb-4c33-8fda-080ef6433b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:28:37.135307 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:37.135283 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhfrh\" (UniqueName: \"kubernetes.io/projected/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-kube-api-access-nhfrh\") pod \"payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv\" (UID: \"ecf97bb5-80cb-4c33-8fda-080ef6433b5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:28:37.225203 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:37.225107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:28:37.579872 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:37.579846 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv"] Jul 24 19:28:37.581967 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:28:37.581942 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf97bb5_80cb_4c33_8fda_080ef6433b5d.slice/crio-0d0e657d03abc2cdc6f053368f7cddc80e4684cc418b2ef4e4576c77d196fa8f WatchSource:0}: Error finding container 0d0e657d03abc2cdc6f053368f7cddc80e4684cc418b2ef4e4576c77d196fa8f: Status 404 returned error can't find the container with id 0d0e657d03abc2cdc6f053368f7cddc80e4684cc418b2ef4e4576c77d196fa8f Jul 24 19:28:38.046417 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:38.046383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" event={"ID":"ecf97bb5-80cb-4c33-8fda-080ef6433b5d","Type":"ContainerStarted","Data":"13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004"} Jul 24 19:28:38.046839 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:38.046433 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" event={"ID":"ecf97bb5-80cb-4c33-8fda-080ef6433b5d","Type":"ContainerStarted","Data":"0d0e657d03abc2cdc6f053368f7cddc80e4684cc418b2ef4e4576c77d196fa8f"} Jul 24 19:28:38.046839 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:38.046492 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:28:38.066130 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:38.066082 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" podStartSLOduration=2.066065286 podStartE2EDuration="2.066065286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:28:38.064139873 +0000 UTC m=+2713.140652432" watchObservedRunningTime="2026-07-24 19:28:38.066065286 +0000 UTC m=+2713.142577874" Jul 24 19:28:49.054384 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:49.054351 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:28:57.780617 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:28:57.780585 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vqv6d_be404140-206a-4cb9-b1e2-191c44c91c7f/payload-processing/0.log" Jul 24 19:29:36.661073 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:36.661041 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vqv6d_be404140-206a-4cb9-b1e2-191c44c91c7f/payload-processing/0.log" Jul 24 19:29:40.521503 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.521470 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689"] Jul 24 19:29:40.532192 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.532165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:40.533551 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.533523 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689"] Jul 24 19:29:40.535018 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.534988 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-ad8f3a-serving-cert\"" Jul 24 19:29:40.603062 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.603021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0764b748-184a-44ba-b26d-e69ce05e0305-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:40.603286 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.603078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0764b748-184a-44ba-b26d-e69ce05e0305-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:40.603286 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.603145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmrn\" (UniqueName: \"kubernetes.io/projected/0764b748-184a-44ba-b26d-e69ce05e0305-kube-api-access-kbmrn\") pod \"maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:40.703659 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.703624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0764b748-184a-44ba-b26d-e69ce05e0305-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:40.703659 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.703663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0764b748-184a-44ba-b26d-e69ce05e0305-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:40.703899 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.703695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmrn\" (UniqueName: \"kubernetes.io/projected/0764b748-184a-44ba-b26d-e69ce05e0305-kube-api-access-kbmrn\") pod \"maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:40.703899 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:29:40.703834 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-serving-cert: secret "maas-api-e2e-ipp-cleanup-ad8f3a-serving-cert" not found Jul 24 19:29:40.703980 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:29:40.703918 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0764b748-184a-44ba-b26d-e69ce05e0305-maas-api-tls podName:0764b748-184a-44ba-b26d-e69ce05e0305 nodeName:}" failed. No retries permitted until 2026-07-24 19:29:41.203895019 +0000 UTC m=+2776.280407572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0764b748-184a-44ba-b26d-e69ce05e0305-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" (UID: "0764b748-184a-44ba-b26d-e69ce05e0305") : secret "maas-api-e2e-ipp-cleanup-ad8f3a-serving-cert" not found Jul 24 19:29:40.704457 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.704435 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0764b748-184a-44ba-b26d-e69ce05e0305-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:40.712485 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:40.712456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmrn\" (UniqueName: \"kubernetes.io/projected/0764b748-184a-44ba-b26d-e69ce05e0305-kube-api-access-kbmrn\") pod \"maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:41.206730 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.206691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0764b748-184a-44ba-b26d-e69ce05e0305-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:41.209278 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.209253 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0764b748-184a-44ba-b26d-e69ce05e0305-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:41.282504 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.282469 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5"] Jul 24 19:29:41.286271 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.286252 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:29:41.288720 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.288701 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ad8f3a-dockercfg-v8zd8\"" Jul 24 19:29:41.289011 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.288998 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ad8f3a\"" Jul 24 19:29:41.301922 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.301899 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5"] Jul 24 19:29:41.307889 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.307862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvrr9\" (UniqueName: \"kubernetes.io/projected/97184c3f-d196-4884-b1d2-9cbf27f4c76a-kube-api-access-tvrr9\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5\" (UID: \"97184c3f-d196-4884-b1d2-9cbf27f4c76a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:29:41.308012 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.307923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97184c3f-d196-4884-b1d2-9cbf27f4c76a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5\" (UID: \"97184c3f-d196-4884-b1d2-9cbf27f4c76a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:29:41.349585 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.349549 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr"] Jul 24 19:29:41.353202 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.353184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:29:41.364470 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.364446 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr"] Jul 24 19:29:41.408751 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.408716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97184c3f-d196-4884-b1d2-9cbf27f4c76a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5\" (UID: \"97184c3f-d196-4884-b1d2-9cbf27f4c76a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:29:41.408936 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.408807 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5f5c\" (UniqueName: \"kubernetes.io/projected/0e54207c-50d2-46f4-8f08-4232e6f939f2-kube-api-access-r5f5c\") pod \"payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr\" (UID: \"0e54207c-50d2-46f4-8f08-4232e6f939f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:29:41.408936 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.408831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e54207c-50d2-46f4-8f08-4232e6f939f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr\" (UID: \"0e54207c-50d2-46f4-8f08-4232e6f939f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:29:41.408936 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.408878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvrr9\" (UniqueName: \"kubernetes.io/projected/97184c3f-d196-4884-b1d2-9cbf27f4c76a-kube-api-access-tvrr9\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5\" (UID: \"97184c3f-d196-4884-b1d2-9cbf27f4c76a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:29:41.409464 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.409442 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97184c3f-d196-4884-b1d2-9cbf27f4c76a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5\" (UID: \"97184c3f-d196-4884-b1d2-9cbf27f4c76a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:29:41.418583 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.418549 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvrr9\" (UniqueName: \"kubernetes.io/projected/97184c3f-d196-4884-b1d2-9cbf27f4c76a-kube-api-access-tvrr9\") pod \"payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5\" (UID: \"97184c3f-d196-4884-b1d2-9cbf27f4c76a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:29:41.446637 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.446601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:41.509917 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.509876 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5f5c\" (UniqueName: \"kubernetes.io/projected/0e54207c-50d2-46f4-8f08-4232e6f939f2-kube-api-access-r5f5c\") pod \"payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr\" (UID: \"0e54207c-50d2-46f4-8f08-4232e6f939f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:29:41.510259 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.509935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e54207c-50d2-46f4-8f08-4232e6f939f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr\" (UID: \"0e54207c-50d2-46f4-8f08-4232e6f939f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:29:41.511265 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.511224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e54207c-50d2-46f4-8f08-4232e6f939f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr\" (UID: \"0e54207c-50d2-46f4-8f08-4232e6f939f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:29:41.520033 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.520006 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5f5c\" (UniqueName: \"kubernetes.io/projected/0e54207c-50d2-46f4-8f08-4232e6f939f2-kube-api-access-r5f5c\") pod \"payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr\" (UID: \"0e54207c-50d2-46f4-8f08-4232e6f939f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:29:41.597306 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.597225 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:29:41.603436 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.603410 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689"] Jul 24 19:29:41.610950 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:29:41.610925 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0764b748_184a_44ba_b26d_e69ce05e0305.slice/crio-b5df18615509b5e1f072742bb1d68c3ecf04dd34e68fae1ea41b6ac26bc72ae1 WatchSource:0}: Error finding container b5df18615509b5e1f072742bb1d68c3ecf04dd34e68fae1ea41b6ac26bc72ae1: Status 404 returned error can't find the container with id b5df18615509b5e1f072742bb1d68c3ecf04dd34e68fae1ea41b6ac26bc72ae1 Jul 24 19:29:41.664529 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.664271 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:29:41.967095 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:41.967070 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5"] Jul 24 19:29:41.969373 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:29:41.969344 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97184c3f_d196_4884_b1d2_9cbf27f4c76a.slice/crio-ec8c17a80ae67bebae775ac6b2b7f880215aa9d3f2c5b020aae4fa1a7126b548 WatchSource:0}: Error finding container ec8c17a80ae67bebae775ac6b2b7f880215aa9d3f2c5b020aae4fa1a7126b548: Status 404 returned error can't find the container with id ec8c17a80ae67bebae775ac6b2b7f880215aa9d3f2c5b020aae4fa1a7126b548 Jul 24 19:29:42.036212 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.036173 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr"] Jul 24 19:29:42.038547 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:29:42.038520 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e54207c_50d2_46f4_8f08_4232e6f939f2.slice/crio-394b7c590541103bb4287ed7924e5674b32a642180a3f9ef7ec7d8790a56932f WatchSource:0}: Error finding container 394b7c590541103bb4287ed7924e5674b32a642180a3f9ef7ec7d8790a56932f: Status 404 returned error can't find the container with id 394b7c590541103bb4287ed7924e5674b32a642180a3f9ef7ec7d8790a56932f Jul 24 19:29:42.412547 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.412510 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" event={"ID":"0e54207c-50d2-46f4-8f08-4232e6f939f2","Type":"ContainerStarted","Data":"74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3"} Jul 24 19:29:42.412852 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.412814 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" event={"ID":"0e54207c-50d2-46f4-8f08-4232e6f939f2","Type":"ContainerStarted","Data":"394b7c590541103bb4287ed7924e5674b32a642180a3f9ef7ec7d8790a56932f"} Jul 24 19:29:42.413004 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.412876 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:29:42.414674 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.414566 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" event={"ID":"97184c3f-d196-4884-b1d2-9cbf27f4c76a","Type":"ContainerStarted","Data":"59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc"} Jul 24 19:29:42.414674 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.414617 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" event={"ID":"97184c3f-d196-4884-b1d2-9cbf27f4c76a","Type":"ContainerStarted","Data":"ec8c17a80ae67bebae775ac6b2b7f880215aa9d3f2c5b020aae4fa1a7126b548"} Jul 24 19:29:42.414674 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.414645 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:29:42.416871 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.416843 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" event={"ID":"0764b748-184a-44ba-b26d-e69ce05e0305","Type":"ContainerStarted","Data":"39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3"} Jul 24 19:29:42.416978 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.416890 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" event={"ID":"0764b748-184a-44ba-b26d-e69ce05e0305","Type":"ContainerStarted","Data":"b5df18615509b5e1f072742bb1d68c3ecf04dd34e68fae1ea41b6ac26bc72ae1"} Jul 24 19:29:42.416978 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.416936 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:42.431103 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.431058 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" podStartSLOduration=1.431041732 podStartE2EDuration="1.431041732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:29:42.429532131 +0000 UTC m=+2777.506044690" watchObservedRunningTime="2026-07-24 19:29:42.431041732 +0000 UTC m=+2777.507554289" Jul 24 19:29:42.446019 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.445980 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" podStartSLOduration=1.889732304 podStartE2EDuration="2.445967632s" totalImagesPullingTime="556.235328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:29:42.44510001 +0000 UTC m=+2777.521612567" watchObservedRunningTime="2026-07-24 19:29:42.445967632 +0000 UTC m=+2777.522480189" Jul 24 19:29:42.462997 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:42.462959 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" podStartSLOduration=1.4629472350000001 podStartE2EDuration="1.462947235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:29:42.460918436 +0000 UTC m=+2777.537430993" watchObservedRunningTime="2026-07-24 19:29:42.462947235 +0000 UTC m=+2777.539459810" Jul 24 19:29:48.427966 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:48.427889 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:29:53.424188 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:53.424160 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:29:53.424597 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:53.424227 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:29:55.783789 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:55.783751 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc"] Jul 24 19:29:55.788826 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:55.788810 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc" Jul 24 19:29:55.791809 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:55.791784 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:29:55.794919 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:55.794892 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc"] Jul 24 19:29:55.851903 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:55.851867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bef1a42-5147-4c0b-83f0-7d13820da6ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc\" (UID: \"3bef1a42-5147-4c0b-83f0-7d13820da6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc" Jul 24 19:29:55.952349 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:55.952316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bef1a42-5147-4c0b-83f0-7d13820da6ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc\" (UID: \"3bef1a42-5147-4c0b-83f0-7d13820da6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc" Jul 24 19:29:55.952975 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:55.952956 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bef1a42-5147-4c0b-83f0-7d13820da6ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc\" (UID: \"3bef1a42-5147-4c0b-83f0-7d13820da6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc" Jul 24 19:29:56.103126 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:56.103033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc" Jul 24 19:29:56.280406 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:56.280360 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc"] Jul 24 19:29:56.283260 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:29:56.283235 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bef1a42_5147_4c0b_83f0_7d13820da6ef.slice/crio-d275fbe2c82fe753ede16cb7fe55e8e2e4e352bb2ccd14a9d1f71219683fba79 WatchSource:0}: Error finding container d275fbe2c82fe753ede16cb7fe55e8e2e4e352bb2ccd14a9d1f71219683fba79: Status 404 returned error can't find the container with id d275fbe2c82fe753ede16cb7fe55e8e2e4e352bb2ccd14a9d1f71219683fba79 Jul 24 19:29:56.506546 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:56.506501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc" event={"ID":"3bef1a42-5147-4c0b-83f0-7d13820da6ef","Type":"ContainerStarted","Data":"d275fbe2c82fe753ede16cb7fe55e8e2e4e352bb2ccd14a9d1f71219683fba79"} Jul 24 19:29:57.514456 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:57.514423 2575 generic.go:356] "Generic (PLEG): container finished" podID="3bef1a42-5147-4c0b-83f0-7d13820da6ef" containerID="3326ac4465047909a0ca46f7beecfcd678ed517337e47516456a8103c95b8fe2" exitCode=0 Jul 24 19:29:57.514881 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:57.514516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc" event={"ID":"3bef1a42-5147-4c0b-83f0-7d13820da6ef","Type":"ContainerDied","Data":"3326ac4465047909a0ca46f7beecfcd678ed517337e47516456a8103c95b8fe2"} Jul 24 19:29:58.668493 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:58.668471 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc" Jul 24 19:29:58.781062 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:58.780979 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bef1a42-5147-4c0b-83f0-7d13820da6ef-maas-api-service-ca\") pod \"3bef1a42-5147-4c0b-83f0-7d13820da6ef\" (UID: \"3bef1a42-5147-4c0b-83f0-7d13820da6ef\") " Jul 24 19:29:58.781321 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:58.781297 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bef1a42-5147-4c0b-83f0-7d13820da6ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bef1a42-5147-4c0b-83f0-7d13820da6ef" (UID: "3bef1a42-5147-4c0b-83f0-7d13820da6ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:29:58.882355 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:58.882314 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bef1a42-5147-4c0b-83f0-7d13820da6ef-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:29:59.526977 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:59.526946 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc" Jul 24 19:29:59.526977 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:59.526967 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc" event={"ID":"3bef1a42-5147-4c0b-83f0-7d13820da6ef","Type":"ContainerDied","Data":"d275fbe2c82fe753ede16cb7fe55e8e2e4e352bb2ccd14a9d1f71219683fba79"} Jul 24 19:29:59.527198 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:29:59.527020 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d275fbe2c82fe753ede16cb7fe55e8e2e4e352bb2ccd14a9d1f71219683fba79" Jul 24 19:30:00.136148 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.136112 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp"] Jul 24 19:30:00.136728 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.136704 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bef1a42-5147-4c0b-83f0-7d13820da6ef" containerName="revoke-keys" Jul 24 19:30:00.136903 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.136762 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bef1a42-5147-4c0b-83f0-7d13820da6ef" containerName="revoke-keys" Jul 24 19:30:00.136972 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.136952 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bef1a42-5147-4c0b-83f0-7d13820da6ef" containerName="revoke-keys" Jul 24 19:30:00.140088 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.140069 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r"] Jul 24 19:30:00.140232 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.140191 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp" Jul 24 19:30:00.143646 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.143619 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r" Jul 24 19:30:00.144379 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.144360 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:30:00.145865 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.145832 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276"] Jul 24 19:30:00.160042 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.160008 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb"] Jul 24 19:30:00.162015 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.161710 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276" Jul 24 19:30:00.170272 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.170245 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp"] Jul 24 19:30:00.171833 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.171811 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp" Jul 24 19:30:00.173090 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.173063 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb" Jul 24 19:30:00.175579 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.175561 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp"] Jul 24 19:30:00.175717 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.175698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp" Jul 24 19:30:00.175975 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.175711 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r"] Jul 24 19:30:00.176129 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.176115 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb"] Jul 24 19:30:00.178469 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.178450 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276"] Jul 24 19:30:00.184261 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.184232 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89"] Jul 24 19:30:00.188510 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.188485 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r" Jul 24 19:30:00.192995 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.192502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp"] Jul 24 19:30:00.192995 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.192639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89" Jul 24 19:30:00.194711 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.194359 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89"] Jul 24 19:30:00.241832 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.241789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276" Jul 24 19:30:00.255022 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.253306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb" Jul 24 19:30:00.289212 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.288760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp" Jul 24 19:30:00.302521 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.302344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89" Jul 24 19:30:00.410452 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:30:00.410344 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f0d94b_75dc_4918_918f_85487d39ba40.slice/crio-5db95a1e8af5e95e6f895ba7a3750e294f7a11b809a18d7600f262fa357cc885 WatchSource:0}: Error finding container 5db95a1e8af5e95e6f895ba7a3750e294f7a11b809a18d7600f262fa357cc885: Status 404 returned error can't find the container with id 5db95a1e8af5e95e6f895ba7a3750e294f7a11b809a18d7600f262fa357cc885 Jul 24 19:30:00.424274 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.424133 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp"] Jul 24 19:30:00.479720 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.479536 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r"] Jul 24 19:30:00.511194 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.511166 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276"] Jul 24 19:30:00.536030 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.535992 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r" event={"ID":"457b53d9-76d6-4a7b-a46e-7fda2ec4a647","Type":"ContainerStarted","Data":"ba89b49a288f84c2fda0013ba35f0efb626c4a29501cb5e073b9ff3975f00e1d"} Jul 24 19:30:00.538445 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.538413 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp" event={"ID":"d6f0d94b-75dc-4918-918f-85487d39ba40","Type":"ContainerStarted","Data":"5db95a1e8af5e95e6f895ba7a3750e294f7a11b809a18d7600f262fa357cc885"} Jul 24 19:30:00.542473 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.542440 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb"] Jul 24 19:30:00.605950 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.605925 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89"] Jul 24 19:30:00.778397 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:00.778361 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp"] Jul 24 19:30:00.780041 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:30:00.780014 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be4e3d3_4154_4901_8b3b_3ca6aa9db22a.slice/crio-e817bd7286d98447a28c56fb2045a7601614a60be231991a45db17f699c31953 WatchSource:0}: Error finding container e817bd7286d98447a28c56fb2045a7601614a60be231991a45db17f699c31953: Status 404 returned error can't find the container with id e817bd7286d98447a28c56fb2045a7601614a60be231991a45db17f699c31953 Jul 24 19:30:01.547261 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.547222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89" event={"ID":"88b72634-10b9-4a12-a08e-270a054e04cf","Type":"ContainerStarted","Data":"7b8e145515a414857c4344013f411987929584a32c28bb9326432f035b39390e"} Jul 24 19:30:01.547699 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.547280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89" event={"ID":"88b72634-10b9-4a12-a08e-270a054e04cf","Type":"ContainerStarted","Data":"a94f0336caed1f8f5a26c1ad0eb4d99f76a2fa9a70495e603720a793064fef1a"} Jul 24 19:30:01.548766 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.548708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r" event={"ID":"457b53d9-76d6-4a7b-a46e-7fda2ec4a647","Type":"ContainerStarted","Data":"44184f14773f4109e5697e3c3d34fe48b3eafb6ce7254abcae92d47d7ecbdf11"} Jul 24 19:30:01.550233 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.550206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276" event={"ID":"90a2882c-af10-480a-8d4b-8e1c54575581","Type":"ContainerStarted","Data":"b6df75715ee92e6adbb559969fc59353af0ca44b22385b82f859a1db723b580f"} Jul 24 19:30:01.550347 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.550260 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276" event={"ID":"90a2882c-af10-480a-8d4b-8e1c54575581","Type":"ContainerStarted","Data":"63fb03c3f67891c83485a46ccf7f08cd8fa6b2fe00bb77add14d609e692230d8"} Jul 24 19:30:01.551626 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.551598 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb" event={"ID":"5d2e0b16-129f-4ba5-91b5-3ba79dcea4ce","Type":"ContainerStarted","Data":"ad0dd2009761b072257a38e86464f50a396237318ec42eba1bd286ede76ab2b5"} Jul 24 19:30:01.551755 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.551646 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb" event={"ID":"5d2e0b16-129f-4ba5-91b5-3ba79dcea4ce","Type":"ContainerStarted","Data":"c0fe8ec9a379c9f3b214f11fed580d8aac781858dd6e85ef77f65f74a6b24c90"} Jul 24 19:30:01.553113 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.553090 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp" event={"ID":"d6f0d94b-75dc-4918-918f-85487d39ba40","Type":"ContainerStarted","Data":"b893b1372486e267286da5bfc045569de8a7850ed73e9574a3126336869e0f5b"} Jul 24 19:30:01.558136 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.558117 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp" event={"ID":"8be4e3d3-4154-4901-8b3b-3ca6aa9db22a","Type":"ContainerStarted","Data":"ebcfd6a0dbbee5ec54f53838791dca1c464e353b489b067746e8ff809f72f1e9"} Jul 24 19:30:01.558224 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.558155 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp" event={"ID":"8be4e3d3-4154-4901-8b3b-3ca6aa9db22a","Type":"ContainerStarted","Data":"e817bd7286d98447a28c56fb2045a7601614a60be231991a45db17f699c31953"} Jul 24 19:30:01.563590 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.563552 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89" podStartSLOduration=1.5635384060000002 podStartE2EDuration="1.563538406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:30:01.563074095 +0000 UTC m=+2796.639586668" watchObservedRunningTime="2026-07-24 19:30:01.563538406 +0000 UTC m=+2796.640050999" Jul 24 19:30:01.580729 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.580680 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp" podStartSLOduration=1.580667984 podStartE2EDuration="1.580667984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:30:01.57798605 +0000 UTC m=+2796.654498610" watchObservedRunningTime="2026-07-24 19:30:01.580667984 +0000 UTC m=+2796.657180542" Jul 24 19:30:01.595525 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.594196 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r" podStartSLOduration=1.5941802520000001 podStartE2EDuration="1.594180252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:30:01.593733305 +0000 UTC m=+2796.670245864" watchObservedRunningTime="2026-07-24 19:30:01.594180252 +0000 UTC m=+2796.670692811" Jul 24 19:30:01.610100 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.610057 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276" podStartSLOduration=1.610042296 podStartE2EDuration="1.610042296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:30:01.608439794 +0000 UTC m=+2796.684952352" watchObservedRunningTime="2026-07-24 19:30:01.610042296 +0000 UTC m=+2796.686554853" Jul 24 19:30:01.622653 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.622607 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb" podStartSLOduration=1.6225917760000002 podStartE2EDuration="1.622591776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:30:01.621989287 +0000 UTC m=+2796.698501848" watchObservedRunningTime="2026-07-24 19:30:01.622591776 +0000 UTC m=+2796.699104383" Jul 24 19:30:01.638305 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:01.638259 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp" podStartSLOduration=1.638245753 podStartE2EDuration="1.638245753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:30:01.636047873 +0000 UTC m=+2796.712560432" watchObservedRunningTime="2026-07-24 19:30:01.638245753 +0000 UTC m=+2796.714758514" Jul 24 19:30:02.570977 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.570942 2575 generic.go:356] "Generic (PLEG): container finished" podID="5d2e0b16-129f-4ba5-91b5-3ba79dcea4ce" containerID="ad0dd2009761b072257a38e86464f50a396237318ec42eba1bd286ede76ab2b5" exitCode=0 Jul 24 19:30:02.571444 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.571028 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb" event={"ID":"5d2e0b16-129f-4ba5-91b5-3ba79dcea4ce","Type":"ContainerDied","Data":"ad0dd2009761b072257a38e86464f50a396237318ec42eba1bd286ede76ab2b5"} Jul 24 19:30:02.572665 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.572625 2575 generic.go:356] "Generic (PLEG): container finished" podID="d6f0d94b-75dc-4918-918f-85487d39ba40" containerID="b893b1372486e267286da5bfc045569de8a7850ed73e9574a3126336869e0f5b" exitCode=0 Jul 24 19:30:02.572833 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.572715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp" event={"ID":"d6f0d94b-75dc-4918-918f-85487d39ba40","Type":"ContainerDied","Data":"b893b1372486e267286da5bfc045569de8a7850ed73e9574a3126336869e0f5b"} Jul 24 19:30:02.574502 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.574477 2575 generic.go:356] "Generic (PLEG): container finished" podID="8be4e3d3-4154-4901-8b3b-3ca6aa9db22a" containerID="ebcfd6a0dbbee5ec54f53838791dca1c464e353b489b067746e8ff809f72f1e9" exitCode=0 Jul 24 19:30:02.574630 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.574553 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp" event={"ID":"8be4e3d3-4154-4901-8b3b-3ca6aa9db22a","Type":"ContainerDied","Data":"ebcfd6a0dbbee5ec54f53838791dca1c464e353b489b067746e8ff809f72f1e9"} Jul 24 19:30:02.576212 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.576195 2575 generic.go:356] "Generic (PLEG): container finished" podID="88b72634-10b9-4a12-a08e-270a054e04cf" containerID="7b8e145515a414857c4344013f411987929584a32c28bb9326432f035b39390e" exitCode=0 Jul 24 19:30:02.576305 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.576268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89" event={"ID":"88b72634-10b9-4a12-a08e-270a054e04cf","Type":"ContainerDied","Data":"7b8e145515a414857c4344013f411987929584a32c28bb9326432f035b39390e"} Jul 24 19:30:02.577600 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.577582 2575 generic.go:356] "Generic (PLEG): container finished" podID="457b53d9-76d6-4a7b-a46e-7fda2ec4a647" containerID="44184f14773f4109e5697e3c3d34fe48b3eafb6ce7254abcae92d47d7ecbdf11" exitCode=0 Jul 24 19:30:02.577682 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.577636 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r" event={"ID":"457b53d9-76d6-4a7b-a46e-7fda2ec4a647","Type":"ContainerDied","Data":"44184f14773f4109e5697e3c3d34fe48b3eafb6ce7254abcae92d47d7ecbdf11"} Jul 24 19:30:02.579073 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.579053 2575 generic.go:356] "Generic (PLEG): container finished" podID="90a2882c-af10-480a-8d4b-8e1c54575581" containerID="b6df75715ee92e6adbb559969fc59353af0ca44b22385b82f859a1db723b580f" exitCode=0 Jul 24 19:30:02.579166 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:02.579080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276" event={"ID":"90a2882c-af10-480a-8d4b-8e1c54575581","Type":"ContainerDied","Data":"b6df75715ee92e6adbb559969fc59353af0ca44b22385b82f859a1db723b580f"} Jul 24 19:30:03.770635 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:03.770437 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r" Jul 24 19:30:03.870332 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:03.870301 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89" Jul 24 19:30:03.872505 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:03.872409 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp" Jul 24 19:30:03.875116 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:03.875090 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276" Jul 24 19:30:03.879284 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:03.879264 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb" Jul 24 19:30:03.883435 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:03.883412 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp" Jul 24 19:30:04.592461 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.592429 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89" Jul 24 19:30:04.592461 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.592455 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748690-tdx89" event={"ID":"88b72634-10b9-4a12-a08e-270a054e04cf","Type":"ContainerDied","Data":"a94f0336caed1f8f5a26c1ad0eb4d99f76a2fa9a70495e603720a793064fef1a"} Jul 24 19:30:04.592700 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.592508 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a94f0336caed1f8f5a26c1ad0eb4d99f76a2fa9a70495e603720a793064fef1a" Jul 24 19:30:04.594032 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.594002 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r" Jul 24 19:30:04.594192 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.594004 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r" event={"ID":"457b53d9-76d6-4a7b-a46e-7fda2ec4a647","Type":"ContainerDied","Data":"ba89b49a288f84c2fda0013ba35f0efb626c4a29501cb5e073b9ff3975f00e1d"} Jul 24 19:30:04.594192 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.594135 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba89b49a288f84c2fda0013ba35f0efb626c4a29501cb5e073b9ff3975f00e1d" Jul 24 19:30:04.599812 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.595782 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276" Jul 24 19:30:04.599812 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.595779 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276" event={"ID":"90a2882c-af10-480a-8d4b-8e1c54575581","Type":"ContainerDied","Data":"63fb03c3f67891c83485a46ccf7f08cd8fa6b2fe00bb77add14d609e692230d8"} Jul 24 19:30:04.599812 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.596241 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63fb03c3f67891c83485a46ccf7f08cd8fa6b2fe00bb77add14d609e692230d8" Jul 24 19:30:04.599812 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.597280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb" event={"ID":"5d2e0b16-129f-4ba5-91b5-3ba79dcea4ce","Type":"ContainerDied","Data":"c0fe8ec9a379c9f3b214f11fed580d8aac781858dd6e85ef77f65f74a6b24c90"} Jul 24 19:30:04.599812 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.597319 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0fe8ec9a379c9f3b214f11fed580d8aac781858dd6e85ef77f65f74a6b24c90" Jul 24 19:30:04.599812 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.597375 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb" Jul 24 19:30:04.601129 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.601099 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp" event={"ID":"d6f0d94b-75dc-4918-918f-85487d39ba40","Type":"ContainerDied","Data":"5db95a1e8af5e95e6f895ba7a3750e294f7a11b809a18d7600f262fa357cc885"} Jul 24 19:30:04.601260 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.601146 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5db95a1e8af5e95e6f895ba7a3750e294f7a11b809a18d7600f262fa357cc885" Jul 24 19:30:04.601260 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.601199 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp" Jul 24 19:30:04.603067 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.603046 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp" event={"ID":"8be4e3d3-4154-4901-8b3b-3ca6aa9db22a","Type":"ContainerDied","Data":"e817bd7286d98447a28c56fb2045a7601614a60be231991a45db17f699c31953"} Jul 24 19:30:04.603162 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.603088 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e817bd7286d98447a28c56fb2045a7601614a60be231991a45db17f699c31953" Jul 24 19:30:04.603162 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:04.603137 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp" Jul 24 19:30:07.105336 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.105305 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689"] Jul 24 19:30:07.105825 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.105542 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" podUID="0764b748-184a-44ba-b26d-e69ce05e0305" containerName="maas-api" containerID="cri-o://39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3" gracePeriod=30 Jul 24 19:30:07.142419 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.142389 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp"] Jul 24 19:30:07.144783 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.144762 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-cleanup-ad8f3a-29748690-7n4mp"] Jul 24 19:30:07.172870 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.172834 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr"] Jul 24 19:30:07.173160 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.173088 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" podUID="0e54207c-50d2-46f4-8f08-4232e6f939f2" containerName="payload-processing" containerID="cri-o://74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3" gracePeriod=30 Jul 24 19:30:07.181206 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.181032 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5"] Jul 24 19:30:07.181322 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.181283 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" podUID="97184c3f-d196-4884-b1d2-9cbf27f4c76a" containerName="payload-pre-processing" containerID="cri-o://59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc" gracePeriod=30 Jul 24 19:30:07.422647 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.422615 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6f0d94b-75dc-4918-918f-85487d39ba40" path="/var/lib/kubelet/pods/d6f0d94b-75dc-4918-918f-85487d39ba40/volumes" Jul 24 19:30:07.497550 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.497235 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:30:07.518861 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.518835 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:30:07.523150 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.523129 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:30:07.574758 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.574711 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5f5c\" (UniqueName: \"kubernetes.io/projected/0e54207c-50d2-46f4-8f08-4232e6f939f2-kube-api-access-r5f5c\") pod \"0e54207c-50d2-46f4-8f08-4232e6f939f2\" (UID: \"0e54207c-50d2-46f4-8f08-4232e6f939f2\") " Jul 24 19:30:07.574997 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.574907 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e54207c-50d2-46f4-8f08-4232e6f939f2-plugins-config-volume\") pod \"0e54207c-50d2-46f4-8f08-4232e6f939f2\" (UID: \"0e54207c-50d2-46f4-8f08-4232e6f939f2\") " Jul 24 19:30:07.575269 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.575244 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e54207c-50d2-46f4-8f08-4232e6f939f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0e54207c-50d2-46f4-8f08-4232e6f939f2" (UID: "0e54207c-50d2-46f4-8f08-4232e6f939f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:30:07.576864 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.576843 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e54207c-50d2-46f4-8f08-4232e6f939f2-kube-api-access-r5f5c" (OuterVolumeSpecName: "kube-api-access-r5f5c") pod "0e54207c-50d2-46f4-8f08-4232e6f939f2" (UID: "0e54207c-50d2-46f4-8f08-4232e6f939f2"). InnerVolumeSpecName "kube-api-access-r5f5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:30:07.622344 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.622308 2575 generic.go:356] "Generic (PLEG): container finished" podID="0e54207c-50d2-46f4-8f08-4232e6f939f2" containerID="74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3" exitCode=0 Jul 24 19:30:07.622628 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.622375 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" Jul 24 19:30:07.622628 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.622390 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" event={"ID":"0e54207c-50d2-46f4-8f08-4232e6f939f2","Type":"ContainerDied","Data":"74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3"} Jul 24 19:30:07.622628 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.622461 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr" event={"ID":"0e54207c-50d2-46f4-8f08-4232e6f939f2","Type":"ContainerDied","Data":"394b7c590541103bb4287ed7924e5674b32a642180a3f9ef7ec7d8790a56932f"} Jul 24 19:30:07.622628 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.622490 2575 scope.go:117] "RemoveContainer" containerID="74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3" Jul 24 19:30:07.623777 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.623756 2575 generic.go:356] "Generic (PLEG): container finished" podID="97184c3f-d196-4884-b1d2-9cbf27f4c76a" containerID="59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc" exitCode=0 Jul 24 19:30:07.623875 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.623793 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" event={"ID":"97184c3f-d196-4884-b1d2-9cbf27f4c76a","Type":"ContainerDied","Data":"59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc"} Jul 24 19:30:07.623875 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.623851 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" event={"ID":"97184c3f-d196-4884-b1d2-9cbf27f4c76a","Type":"ContainerDied","Data":"ec8c17a80ae67bebae775ac6b2b7f880215aa9d3f2c5b020aae4fa1a7126b548"} Jul 24 19:30:07.623875 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.623808 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5" Jul 24 19:30:07.625362 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.625339 2575 generic.go:356] "Generic (PLEG): container finished" podID="0764b748-184a-44ba-b26d-e69ce05e0305" containerID="39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3" exitCode=0 Jul 24 19:30:07.625492 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.625383 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" Jul 24 19:30:07.625492 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.625427 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" event={"ID":"0764b748-184a-44ba-b26d-e69ce05e0305","Type":"ContainerDied","Data":"39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3"} Jul 24 19:30:07.625492 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.625475 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689" event={"ID":"0764b748-184a-44ba-b26d-e69ce05e0305","Type":"ContainerDied","Data":"b5df18615509b5e1f072742bb1d68c3ecf04dd34e68fae1ea41b6ac26bc72ae1"} Jul 24 19:30:07.632572 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.632554 2575 scope.go:117] "RemoveContainer" containerID="74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3" Jul 24 19:30:07.632866 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:30:07.632847 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3\": container with ID starting with 74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3 not found: ID does not exist" containerID="74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3" Jul 24 19:30:07.632926 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.632875 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3"} err="failed to get container status \"74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3\": rpc error: code = NotFound desc = could not find container \"74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3\": container with ID starting with 74cd9df3cf066448e8cdfa192fcd77cfa53d1d9513e345f2bade6fb91721c3b3 not found: ID does not exist" Jul 24 19:30:07.632926 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.632891 2575 scope.go:117] "RemoveContainer" containerID="59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc" Jul 24 19:30:07.643863 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.643838 2575 scope.go:117] "RemoveContainer" containerID="59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc" Jul 24 19:30:07.644180 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:30:07.644143 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc\": container with ID starting with 59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc not found: ID does not exist" containerID="59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc" Jul 24 19:30:07.644283 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.644181 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc"} err="failed to get container status \"59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc\": rpc error: code = NotFound desc = could not find container \"59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc\": container with ID starting with 59c9e91c9a0bce7b4444ba428f237035bf072e923af6429821cd7bbb43be06bc not found: ID does not exist" Jul 24 19:30:07.644283 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.644200 2575 scope.go:117] "RemoveContainer" containerID="39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3" Jul 24 19:30:07.647482 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.647465 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr"] Jul 24 19:30:07.651076 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.651057 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ad8f3a-77d87cf44b-xdzmr"] Jul 24 19:30:07.653315 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.653298 2575 scope.go:117] "RemoveContainer" containerID="39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3" Jul 24 19:30:07.653581 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:30:07.653559 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3\": container with ID starting with 39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3 not found: ID does not exist" containerID="39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3" Jul 24 19:30:07.653638 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.653590 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3"} err="failed to get container status \"39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3\": rpc error: code = NotFound desc = could not find container \"39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3\": container with ID starting with 39cd4dab7e42616aacbab5168a06472c6ecd93a0621fe22e476489feec5c02f3 not found: ID does not exist" Jul 24 19:30:07.675914 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.675892 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0764b748-184a-44ba-b26d-e69ce05e0305-maas-api-tls\") pod \"0764b748-184a-44ba-b26d-e69ce05e0305\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " Jul 24 19:30:07.676013 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.675978 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbmrn\" (UniqueName: \"kubernetes.io/projected/0764b748-184a-44ba-b26d-e69ce05e0305-kube-api-access-kbmrn\") pod \"0764b748-184a-44ba-b26d-e69ce05e0305\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " Jul 24 19:30:07.676054 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.676038 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0764b748-184a-44ba-b26d-e69ce05e0305-openshift-service-ca\") pod \"0764b748-184a-44ba-b26d-e69ce05e0305\" (UID: \"0764b748-184a-44ba-b26d-e69ce05e0305\") " Jul 24 19:30:07.676099 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.676086 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97184c3f-d196-4884-b1d2-9cbf27f4c76a-plugins-config-volume\") pod \"97184c3f-d196-4884-b1d2-9cbf27f4c76a\" (UID: \"97184c3f-d196-4884-b1d2-9cbf27f4c76a\") " Jul 24 19:30:07.676238 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.676221 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvrr9\" (UniqueName: \"kubernetes.io/projected/97184c3f-d196-4884-b1d2-9cbf27f4c76a-kube-api-access-tvrr9\") pod \"97184c3f-d196-4884-b1d2-9cbf27f4c76a\" (UID: \"97184c3f-d196-4884-b1d2-9cbf27f4c76a\") " Jul 24 19:30:07.676431 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.676411 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0764b748-184a-44ba-b26d-e69ce05e0305-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0764b748-184a-44ba-b26d-e69ce05e0305" (UID: "0764b748-184a-44ba-b26d-e69ce05e0305"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:30:07.676506 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.676430 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97184c3f-d196-4884-b1d2-9cbf27f4c76a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97184c3f-d196-4884-b1d2-9cbf27f4c76a" (UID: "97184c3f-d196-4884-b1d2-9cbf27f4c76a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:30:07.676597 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.676581 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0764b748-184a-44ba-b26d-e69ce05e0305-openshift-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:30:07.676658 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.676604 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e54207c-50d2-46f4-8f08-4232e6f939f2-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:30:07.676658 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.676618 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97184c3f-d196-4884-b1d2-9cbf27f4c76a-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:30:07.676658 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.676633 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5f5c\" (UniqueName: \"kubernetes.io/projected/0e54207c-50d2-46f4-8f08-4232e6f939f2-kube-api-access-r5f5c\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:30:07.678237 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.678211 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0764b748-184a-44ba-b26d-e69ce05e0305-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0764b748-184a-44ba-b26d-e69ce05e0305" (UID: "0764b748-184a-44ba-b26d-e69ce05e0305"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 19:30:07.678321 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.678264 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0764b748-184a-44ba-b26d-e69ce05e0305-kube-api-access-kbmrn" (OuterVolumeSpecName: "kube-api-access-kbmrn") pod "0764b748-184a-44ba-b26d-e69ce05e0305" (UID: "0764b748-184a-44ba-b26d-e69ce05e0305"). InnerVolumeSpecName "kube-api-access-kbmrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:30:07.678392 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.678373 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97184c3f-d196-4884-b1d2-9cbf27f4c76a-kube-api-access-tvrr9" (OuterVolumeSpecName: "kube-api-access-tvrr9") pod "97184c3f-d196-4884-b1d2-9cbf27f4c76a" (UID: "97184c3f-d196-4884-b1d2-9cbf27f4c76a"). InnerVolumeSpecName "kube-api-access-tvrr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:30:07.778071 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.778039 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvrr9\" (UniqueName: \"kubernetes.io/projected/97184c3f-d196-4884-b1d2-9cbf27f4c76a-kube-api-access-tvrr9\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:30:07.778071 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.778065 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0764b748-184a-44ba-b26d-e69ce05e0305-maas-api-tls\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:30:07.778071 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.778076 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmrn\" (UniqueName: \"kubernetes.io/projected/0764b748-184a-44ba-b26d-e69ce05e0305-kube-api-access-kbmrn\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:30:07.949650 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.949619 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5"] Jul 24 19:30:07.953670 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.953646 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-ad8f3a-754475b6fb-h74s5"] Jul 24 19:30:07.965634 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.965606 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689"] Jul 24 19:30:07.968965 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:07.968943 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ad8f3a-6f5fd7f748-25689"] Jul 24 19:30:09.418299 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:09.418265 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0764b748-184a-44ba-b26d-e69ce05e0305" path="/var/lib/kubelet/pods/0764b748-184a-44ba-b26d-e69ce05e0305/volumes" Jul 24 19:30:09.418673 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:09.418613 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e54207c-50d2-46f4-8f08-4232e6f939f2" path="/var/lib/kubelet/pods/0e54207c-50d2-46f4-8f08-4232e6f939f2/volumes" Jul 24 19:30:09.419144 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:09.419123 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97184c3f-d196-4884-b1d2-9cbf27f4c76a" path="/var/lib/kubelet/pods/97184c3f-d196-4884-b1d2-9cbf27f4c76a/volumes" Jul 24 19:30:10.865073 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:10.865034 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc"] Jul 24 19:30:10.867875 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:10.867849 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ad8f3a-rzvlc"] Jul 24 19:30:11.417439 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:11.417402 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bef1a42-5147-4c0b-83f0-7d13820da6ef" path="/var/lib/kubelet/pods/3bef1a42-5147-4c0b-83f0-7d13820da6ef/volumes" Jul 24 19:30:21.424695 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.424664 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh"] Jul 24 19:30:21.425122 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425108 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e54207c-50d2-46f4-8f08-4232e6f939f2" containerName="payload-processing" Jul 24 19:30:21.425176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425124 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e54207c-50d2-46f4-8f08-4232e6f939f2" containerName="payload-processing" Jul 24 19:30:21.425176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425133 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457b53d9-76d6-4a7b-a46e-7fda2ec4a647" containerName="cleanup" Jul 24 19:30:21.425176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425139 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="457b53d9-76d6-4a7b-a46e-7fda2ec4a647" containerName="cleanup" Jul 24 19:30:21.425176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425153 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0764b748-184a-44ba-b26d-e69ce05e0305" containerName="maas-api" Jul 24 19:30:21.425176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425158 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0764b748-184a-44ba-b26d-e69ce05e0305" containerName="maas-api" Jul 24 19:30:21.425176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425165 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a2882c-af10-480a-8d4b-8e1c54575581" containerName="cleanup" Jul 24 19:30:21.425176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425170 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a2882c-af10-480a-8d4b-8e1c54575581" containerName="cleanup" Jul 24 19:30:21.425176 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425177 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f0d94b-75dc-4918-918f-85487d39ba40" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425182 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f0d94b-75dc-4918-918f-85487d39ba40" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425188 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b72634-10b9-4a12-a08e-270a054e04cf" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425194 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b72634-10b9-4a12-a08e-270a054e04cf" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425204 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8be4e3d3-4154-4901-8b3b-3ca6aa9db22a" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425209 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8be4e3d3-4154-4901-8b3b-3ca6aa9db22a" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425216 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97184c3f-d196-4884-b1d2-9cbf27f4c76a" containerName="payload-pre-processing" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425221 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="97184c3f-d196-4884-b1d2-9cbf27f4c76a" containerName="payload-pre-processing" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425227 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2e0b16-129f-4ba5-91b5-3ba79dcea4ce" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425232 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2e0b16-129f-4ba5-91b5-3ba79dcea4ce" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425305 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="97184c3f-d196-4884-b1d2-9cbf27f4c76a" containerName="payload-pre-processing" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425312 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="90a2882c-af10-480a-8d4b-8e1c54575581" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425318 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8be4e3d3-4154-4901-8b3b-3ca6aa9db22a" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425324 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e54207c-50d2-46f4-8f08-4232e6f939f2" containerName="payload-processing" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425330 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2e0b16-129f-4ba5-91b5-3ba79dcea4ce" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425337 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0764b748-184a-44ba-b26d-e69ce05e0305" containerName="maas-api" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425344 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f0d94b-75dc-4918-918f-85487d39ba40" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425350 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="457b53d9-76d6-4a7b-a46e-7fda2ec4a647" containerName="cleanup" Jul 24 19:30:21.425465 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.425355 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b72634-10b9-4a12-a08e-270a054e04cf" containerName="cleanup" Jul 24 19:30:21.429720 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.429702 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh" Jul 24 19:30:21.432344 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.432316 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:30:21.435797 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.435771 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh"] Jul 24 19:30:21.503539 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.503504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7956bd7c-9ea1-4f6b-9350-23db8b6f0422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh\" (UID: \"7956bd7c-9ea1-4f6b-9350-23db8b6f0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh" Jul 24 19:30:21.604388 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.604353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7956bd7c-9ea1-4f6b-9350-23db8b6f0422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh\" (UID: \"7956bd7c-9ea1-4f6b-9350-23db8b6f0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh" Jul 24 19:30:21.605051 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.605025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7956bd7c-9ea1-4f6b-9350-23db8b6f0422-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh\" (UID: \"7956bd7c-9ea1-4f6b-9350-23db8b6f0422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh" Jul 24 19:30:21.742146 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.742064 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh" Jul 24 19:30:21.887553 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:21.887522 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh"] Jul 24 19:30:21.889609 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:30:21.889587 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7956bd7c_9ea1_4f6b_9350_23db8b6f0422.slice/crio-546b9cca5bd540c0ea66cb5678337fd40c4975f1a570f41bc77d79be3e0405d6 WatchSource:0}: Error finding container 546b9cca5bd540c0ea66cb5678337fd40c4975f1a570f41bc77d79be3e0405d6: Status 404 returned error can't find the container with id 546b9cca5bd540c0ea66cb5678337fd40c4975f1a570f41bc77d79be3e0405d6 Jul 24 19:30:22.719964 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:22.719929 2575 generic.go:356] "Generic (PLEG): container finished" podID="7956bd7c-9ea1-4f6b-9350-23db8b6f0422" containerID="f7fd65ad8c307c8a6b7ff9783328484f02a7321ffc153b6578553e98adf47c25" exitCode=0 Jul 24 19:30:22.720336 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:22.720012 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh" event={"ID":"7956bd7c-9ea1-4f6b-9350-23db8b6f0422","Type":"ContainerDied","Data":"f7fd65ad8c307c8a6b7ff9783328484f02a7321ffc153b6578553e98adf47c25"} Jul 24 19:30:22.720336 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:22.720063 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh" event={"ID":"7956bd7c-9ea1-4f6b-9350-23db8b6f0422","Type":"ContainerStarted","Data":"546b9cca5bd540c0ea66cb5678337fd40c4975f1a570f41bc77d79be3e0405d6"} Jul 24 19:30:23.871110 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:23.871085 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh" Jul 24 19:30:23.925500 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:23.925465 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7956bd7c-9ea1-4f6b-9350-23db8b6f0422-maas-api-service-ca\") pod \"7956bd7c-9ea1-4f6b-9350-23db8b6f0422\" (UID: \"7956bd7c-9ea1-4f6b-9350-23db8b6f0422\") " Jul 24 19:30:23.925875 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:23.925849 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7956bd7c-9ea1-4f6b-9350-23db8b6f0422-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7956bd7c-9ea1-4f6b-9350-23db8b6f0422" (UID: "7956bd7c-9ea1-4f6b-9350-23db8b6f0422"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:30:24.026657 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:24.026572 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7956bd7c-9ea1-4f6b-9350-23db8b6f0422-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:30:24.733332 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:24.733302 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh" Jul 24 19:30:24.733332 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:24.733322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh" event={"ID":"7956bd7c-9ea1-4f6b-9350-23db8b6f0422","Type":"ContainerDied","Data":"546b9cca5bd540c0ea66cb5678337fd40c4975f1a570f41bc77d79be3e0405d6"} Jul 24 19:30:24.733534 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:24.733367 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="546b9cca5bd540c0ea66cb5678337fd40c4975f1a570f41bc77d79be3e0405d6" Jul 24 19:30:37.758124 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:37.758086 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb"] Jul 24 19:30:37.760513 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:37.760481 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-87cd66-29748690-s9qsb"] Jul 24 19:30:39.417860 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:39.417831 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d2e0b16-129f-4ba5-91b5-3ba79dcea4ce" path="/var/lib/kubelet/pods/5d2e0b16-129f-4ba5-91b5-3ba79dcea4ce/volumes" Jul 24 19:30:41.508449 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:41.508407 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh"] Jul 24 19:30:41.511373 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:41.511350 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-87cd66-6w5wh"] Jul 24 19:30:43.417201 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:30:43.417168 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7956bd7c-9ea1-4f6b-9350-23db8b6f0422" path="/var/lib/kubelet/pods/7956bd7c-9ea1-4f6b-9350-23db8b6f0422/volumes" Jul 24 19:31:30.459056 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.459017 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr"] Jul 24 19:31:30.461671 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.459722 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7956bd7c-9ea1-4f6b-9350-23db8b6f0422" containerName="revoke-keys" Jul 24 19:31:30.461671 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.459767 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7956bd7c-9ea1-4f6b-9350-23db8b6f0422" containerName="revoke-keys" Jul 24 19:31:30.461671 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.459888 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7956bd7c-9ea1-4f6b-9350-23db8b6f0422" containerName="revoke-keys" Jul 24 19:31:30.463328 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.463310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr" Jul 24 19:31:30.466283 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.466264 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:31:30.470769 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.470730 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr"] Jul 24 19:31:30.525606 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.525570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020cb468-17ad-4517-8257-a842107ba278-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr\" (UID: \"020cb468-17ad-4517-8257-a842107ba278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr" Jul 24 19:31:30.626700 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.626659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020cb468-17ad-4517-8257-a842107ba278-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr\" (UID: \"020cb468-17ad-4517-8257-a842107ba278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr" Jul 24 19:31:30.627369 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.627345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020cb468-17ad-4517-8257-a842107ba278-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr\" (UID: \"020cb468-17ad-4517-8257-a842107ba278\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr" Jul 24 19:31:30.776196 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:30.776107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr" Jul 24 19:31:31.131864 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:31.131837 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr"] Jul 24 19:31:31.133382 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:31:31.133355 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020cb468_17ad_4517_8257_a842107ba278.slice/crio-a68ee37e07d01aef5b563f178fc1c16783e00352585f5edd77dda840607eb19a WatchSource:0}: Error finding container a68ee37e07d01aef5b563f178fc1c16783e00352585f5edd77dda840607eb19a: Status 404 returned error can't find the container with id a68ee37e07d01aef5b563f178fc1c16783e00352585f5edd77dda840607eb19a Jul 24 19:31:32.125268 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:32.125178 2575 generic.go:356] "Generic (PLEG): container finished" podID="020cb468-17ad-4517-8257-a842107ba278" containerID="3cdf06159a46e1eb0fe8f96e10a7444a2b10a91029cc4893536e558cbd2908f8" exitCode=0 Jul 24 19:31:32.125640 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:32.125255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr" event={"ID":"020cb468-17ad-4517-8257-a842107ba278","Type":"ContainerDied","Data":"3cdf06159a46e1eb0fe8f96e10a7444a2b10a91029cc4893536e558cbd2908f8"} Jul 24 19:31:32.125640 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:32.125311 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr" event={"ID":"020cb468-17ad-4517-8257-a842107ba278","Type":"ContainerStarted","Data":"a68ee37e07d01aef5b563f178fc1c16783e00352585f5edd77dda840607eb19a"} Jul 24 19:31:33.267358 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:33.267333 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr" Jul 24 19:31:33.349998 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:33.349965 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020cb468-17ad-4517-8257-a842107ba278-maas-api-service-ca\") pod \"020cb468-17ad-4517-8257-a842107ba278\" (UID: \"020cb468-17ad-4517-8257-a842107ba278\") " Jul 24 19:31:33.350312 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:33.350288 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020cb468-17ad-4517-8257-a842107ba278-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "020cb468-17ad-4517-8257-a842107ba278" (UID: "020cb468-17ad-4517-8257-a842107ba278"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:31:33.451198 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:33.451165 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/020cb468-17ad-4517-8257-a842107ba278-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:31:34.138621 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:34.138591 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr" Jul 24 19:31:34.138621 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:34.138615 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr" event={"ID":"020cb468-17ad-4517-8257-a842107ba278","Type":"ContainerDied","Data":"a68ee37e07d01aef5b563f178fc1c16783e00352585f5edd77dda840607eb19a"} Jul 24 19:31:34.138867 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:34.138662 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a68ee37e07d01aef5b563f178fc1c16783e00352585f5edd77dda840607eb19a" Jul 24 19:31:41.185966 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.185928 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r"] Jul 24 19:31:41.188352 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.188326 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5ce07c-29748690-mtv5r"] Jul 24 19:31:41.241467 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.240585 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv"] Jul 24 19:31:41.241467 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.240893 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" podUID="ecf97bb5-80cb-4c33-8fda-080ef6433b5d" containerName="payload-pre-processing" containerID="cri-o://13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004" gracePeriod=30 Jul 24 19:31:41.419699 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.419667 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="457b53d9-76d6-4a7b-a46e-7fda2ec4a647" path="/var/lib/kubelet/pods/457b53d9-76d6-4a7b-a46e-7fda2ec4a647/volumes" Jul 24 19:31:41.520388 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.520364 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:31:41.628363 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.628335 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhfrh\" (UniqueName: \"kubernetes.io/projected/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-kube-api-access-nhfrh\") pod \"ecf97bb5-80cb-4c33-8fda-080ef6433b5d\" (UID: \"ecf97bb5-80cb-4c33-8fda-080ef6433b5d\") " Jul 24 19:31:41.628546 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.628474 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-plugins-config-volume\") pod \"ecf97bb5-80cb-4c33-8fda-080ef6433b5d\" (UID: \"ecf97bb5-80cb-4c33-8fda-080ef6433b5d\") " Jul 24 19:31:41.628859 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.628839 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecf97bb5-80cb-4c33-8fda-080ef6433b5d" (UID: "ecf97bb5-80cb-4c33-8fda-080ef6433b5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:31:41.630575 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.630545 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-kube-api-access-nhfrh" (OuterVolumeSpecName: "kube-api-access-nhfrh") pod "ecf97bb5-80cb-4c33-8fda-080ef6433b5d" (UID: "ecf97bb5-80cb-4c33-8fda-080ef6433b5d"). InnerVolumeSpecName "kube-api-access-nhfrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:31:41.729774 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.729724 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:31:41.729774 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:41.729772 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhfrh\" (UniqueName: \"kubernetes.io/projected/ecf97bb5-80cb-4c33-8fda-080ef6433b5d-kube-api-access-nhfrh\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:31:42.189345 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:42.189311 2575 generic.go:356] "Generic (PLEG): container finished" podID="ecf97bb5-80cb-4c33-8fda-080ef6433b5d" containerID="13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004" exitCode=0 Jul 24 19:31:42.189913 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:42.189353 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" Jul 24 19:31:42.189913 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:42.189401 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" event={"ID":"ecf97bb5-80cb-4c33-8fda-080ef6433b5d","Type":"ContainerDied","Data":"13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004"} Jul 24 19:31:42.189913 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:42.189456 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv" event={"ID":"ecf97bb5-80cb-4c33-8fda-080ef6433b5d","Type":"ContainerDied","Data":"0d0e657d03abc2cdc6f053368f7cddc80e4684cc418b2ef4e4576c77d196fa8f"} Jul 24 19:31:42.189913 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:42.189480 2575 scope.go:117] "RemoveContainer" containerID="13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004" Jul 24 19:31:42.200003 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:42.199983 2575 scope.go:117] "RemoveContainer" containerID="13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004" Jul 24 19:31:42.200271 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:31:42.200251 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004\": container with ID starting with 13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004 not found: ID does not exist" containerID="13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004" Jul 24 19:31:42.200349 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:42.200278 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004"} err="failed to get container status \"13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004\": rpc error: code = NotFound desc = could not find container \"13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004\": container with ID starting with 13c2b91a9a031566d6bee4bf2218ad411fb01986c2bd3f2fb2f208caa6b56004 not found: ID does not exist" Jul 24 19:31:42.212260 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:42.212233 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv"] Jul 24 19:31:42.215405 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:42.215383 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5ce07c-66d8b6999b-blndv"] Jul 24 19:31:43.417936 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:43.417907 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecf97bb5-80cb-4c33-8fda-080ef6433b5d" path="/var/lib/kubelet/pods/ecf97bb5-80cb-4c33-8fda-080ef6433b5d/volumes" Jul 24 19:31:45.536126 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:45.536085 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr"] Jul 24 19:31:45.538890 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:45.538855 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5ce07c-8qldr"] Jul 24 19:31:47.417057 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:47.417022 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020cb468-17ad-4517-8257-a842107ba278" path="/var/lib/kubelet/pods/020cb468-17ad-4517-8257-a842107ba278/volumes" Jul 24 19:31:52.593817 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.593776 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2"] Jul 24 19:31:52.594447 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.594429 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020cb468-17ad-4517-8257-a842107ba278" containerName="revoke-keys" Jul 24 19:31:52.594499 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.594452 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="020cb468-17ad-4517-8257-a842107ba278" containerName="revoke-keys" Jul 24 19:31:52.594540 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.594502 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf97bb5-80cb-4c33-8fda-080ef6433b5d" containerName="payload-pre-processing" Jul 24 19:31:52.594540 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.594511 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf97bb5-80cb-4c33-8fda-080ef6433b5d" containerName="payload-pre-processing" Jul 24 19:31:52.594633 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.594621 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="020cb468-17ad-4517-8257-a842107ba278" containerName="revoke-keys" Jul 24 19:31:52.594668 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.594640 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf97bb5-80cb-4c33-8fda-080ef6433b5d" containerName="payload-pre-processing" Jul 24 19:31:52.598784 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.598732 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2" Jul 24 19:31:52.602265 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.602237 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:31:52.604229 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.604165 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2"] Jul 24 19:31:52.730268 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.730230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e7a35f-1411-4b01-be2c-7330d8557315-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2\" (UID: \"a8e7a35f-1411-4b01-be2c-7330d8557315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2" Jul 24 19:31:52.830957 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.830921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e7a35f-1411-4b01-be2c-7330d8557315-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2\" (UID: \"a8e7a35f-1411-4b01-be2c-7330d8557315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2" Jul 24 19:31:52.831539 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.831521 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e7a35f-1411-4b01-be2c-7330d8557315-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2\" (UID: \"a8e7a35f-1411-4b01-be2c-7330d8557315\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2" Jul 24 19:31:52.914430 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:52.914398 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2" Jul 24 19:31:53.060128 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:53.060084 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2"] Jul 24 19:31:53.063164 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:31:53.063136 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e7a35f_1411_4b01_be2c_7330d8557315.slice/crio-cc04750a31cd809a89a76287ed5177f415543a2a017dba0b5d78438794fe6827 WatchSource:0}: Error finding container cc04750a31cd809a89a76287ed5177f415543a2a017dba0b5d78438794fe6827: Status 404 returned error can't find the container with id cc04750a31cd809a89a76287ed5177f415543a2a017dba0b5d78438794fe6827 Jul 24 19:31:53.258242 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:53.258160 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2" event={"ID":"a8e7a35f-1411-4b01-be2c-7330d8557315","Type":"ContainerStarted","Data":"cc04750a31cd809a89a76287ed5177f415543a2a017dba0b5d78438794fe6827"} Jul 24 19:31:54.266033 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:54.265993 2575 generic.go:356] "Generic (PLEG): container finished" podID="a8e7a35f-1411-4b01-be2c-7330d8557315" containerID="408eca81cefec599dda913756016ad7e6148445eb8c1437561315fc75d8dac2e" exitCode=0 Jul 24 19:31:54.266443 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:54.266081 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2" event={"ID":"a8e7a35f-1411-4b01-be2c-7330d8557315","Type":"ContainerDied","Data":"408eca81cefec599dda913756016ad7e6148445eb8c1437561315fc75d8dac2e"} Jul 24 19:31:55.426418 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:55.426389 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2" Jul 24 19:31:55.554468 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:55.554378 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e7a35f-1411-4b01-be2c-7330d8557315-maas-api-service-ca\") pod \"a8e7a35f-1411-4b01-be2c-7330d8557315\" (UID: \"a8e7a35f-1411-4b01-be2c-7330d8557315\") " Jul 24 19:31:55.554716 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:55.554691 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e7a35f-1411-4b01-be2c-7330d8557315-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8e7a35f-1411-4b01-be2c-7330d8557315" (UID: "a8e7a35f-1411-4b01-be2c-7330d8557315"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:31:55.655379 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:55.655349 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8e7a35f-1411-4b01-be2c-7330d8557315-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:31:56.279866 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:56.279836 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2" Jul 24 19:31:56.280037 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:56.279873 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2" event={"ID":"a8e7a35f-1411-4b01-be2c-7330d8557315","Type":"ContainerDied","Data":"cc04750a31cd809a89a76287ed5177f415543a2a017dba0b5d78438794fe6827"} Jul 24 19:31:56.280037 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:31:56.279917 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc04750a31cd809a89a76287ed5177f415543a2a017dba0b5d78438794fe6827" Jul 24 19:32:03.046732 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.046695 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp"] Jul 24 19:32:03.051027 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.050994 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e05f4d-29748690-df7tp"] Jul 24 19:32:03.085147 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.085104 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw"] Jul 24 19:32:03.085427 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.085390 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" podUID="1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1" containerName="payload-pre-processing" containerID="cri-o://aa4ea5599808226503a26b779597332b184657bdfbd88421712a88b8f394a32f" gracePeriod=30 Jul 24 19:32:03.323322 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.323234 2575 generic.go:356] "Generic (PLEG): container finished" podID="1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1" containerID="aa4ea5599808226503a26b779597332b184657bdfbd88421712a88b8f394a32f" exitCode=0 Jul 24 19:32:03.323322 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.323297 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" event={"ID":"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1","Type":"ContainerDied","Data":"aa4ea5599808226503a26b779597332b184657bdfbd88421712a88b8f394a32f"} Jul 24 19:32:03.357684 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.357662 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:32:03.418940 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.418908 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8be4e3d3-4154-4901-8b3b-3ca6aa9db22a" path="/var/lib/kubelet/pods/8be4e3d3-4154-4901-8b3b-3ca6aa9db22a/volumes" Jul 24 19:32:03.522631 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.522599 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-plugins-config-volume\") pod \"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1\" (UID: \"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1\") " Jul 24 19:32:03.522846 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.522707 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxh6j\" (UniqueName: \"kubernetes.io/projected/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-kube-api-access-lxh6j\") pod \"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1\" (UID: \"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1\") " Jul 24 19:32:03.523075 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.523046 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1" (UID: "1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:32:03.525017 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.524987 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-kube-api-access-lxh6j" (OuterVolumeSpecName: "kube-api-access-lxh6j") pod "1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1" (UID: "1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1"). InnerVolumeSpecName "kube-api-access-lxh6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:32:03.623582 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.623504 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxh6j\" (UniqueName: \"kubernetes.io/projected/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-kube-api-access-lxh6j\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:32:03.623582 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:03.623534 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:32:04.330099 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:04.330067 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" Jul 24 19:32:04.330099 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:04.330079 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw" event={"ID":"1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1","Type":"ContainerDied","Data":"b8da0e58835e950edc43774ea6751a98cd2a34f7a13e9384c447a97735b65732"} Jul 24 19:32:04.330601 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:04.330161 2575 scope.go:117] "RemoveContainer" containerID="aa4ea5599808226503a26b779597332b184657bdfbd88421712a88b8f394a32f" Jul 24 19:32:04.352643 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:04.352616 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw"] Jul 24 19:32:04.354769 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:04.354732 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-e05f4d-6b4b99d75b-fkmdw"] Jul 24 19:32:05.416880 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:05.416836 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1" path="/var/lib/kubelet/pods/1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1/volumes" Jul 24 19:32:07.673866 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:07.673836 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2"] Jul 24 19:32:07.675921 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:07.675902 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-e05f4d-rrns2"] Jul 24 19:32:09.418040 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:09.418008 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8e7a35f-1411-4b01-be2c-7330d8557315" path="/var/lib/kubelet/pods/a8e7a35f-1411-4b01-be2c-7330d8557315/volumes" Jul 24 19:32:14.398735 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.398699 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7"] Jul 24 19:32:14.399396 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.399373 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e7a35f-1411-4b01-be2c-7330d8557315" containerName="revoke-keys" Jul 24 19:32:14.399473 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.399399 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e7a35f-1411-4b01-be2c-7330d8557315" containerName="revoke-keys" Jul 24 19:32:14.399473 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.399450 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1" containerName="payload-pre-processing" Jul 24 19:32:14.399473 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.399458 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1" containerName="payload-pre-processing" Jul 24 19:32:14.399634 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.399591 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8e7a35f-1411-4b01-be2c-7330d8557315" containerName="revoke-keys" Jul 24 19:32:14.399634 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.399614 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad20cdd-cc9b-4274-afeb-0e7af63f5ce1" containerName="payload-pre-processing" Jul 24 19:32:14.404568 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.404545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" Jul 24 19:32:14.407152 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.407127 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6gvkz\"" Jul 24 19:32:14.413635 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.413606 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7"] Jul 24 19:32:14.520589 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.520552 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d8e53a-a447-4aad-80ae-2447e70dcbb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7\" (UID: \"a4d8e53a-a447-4aad-80ae-2447e70dcbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" Jul 24 19:32:14.621957 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.621924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d8e53a-a447-4aad-80ae-2447e70dcbb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7\" (UID: \"a4d8e53a-a447-4aad-80ae-2447e70dcbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" Jul 24 19:32:14.622584 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.622558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d8e53a-a447-4aad-80ae-2447e70dcbb7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7\" (UID: \"a4d8e53a-a447-4aad-80ae-2447e70dcbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" Jul 24 19:32:14.727330 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.727240 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" Jul 24 19:32:14.872519 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:14.872484 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7"] Jul 24 19:32:14.876009 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:32:14.875981 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d8e53a_a447_4aad_80ae_2447e70dcbb7.slice/crio-2c52a8ed3dc0cd33859f81b11125614ca3c3a005d200068df8b2b4b4305c7083 WatchSource:0}: Error finding container 2c52a8ed3dc0cd33859f81b11125614ca3c3a005d200068df8b2b4b4305c7083: Status 404 returned error can't find the container with id 2c52a8ed3dc0cd33859f81b11125614ca3c3a005d200068df8b2b4b4305c7083 Jul 24 19:32:15.406074 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:15.406038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" event={"ID":"a4d8e53a-a447-4aad-80ae-2447e70dcbb7","Type":"ContainerStarted","Data":"0ea92e8a0712e6a3d9b8f7a4ac83d9813e4962f0942efecca24f65e83f37bb42"} Jul 24 19:32:15.406444 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:15.406096 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" event={"ID":"a4d8e53a-a447-4aad-80ae-2447e70dcbb7","Type":"ContainerStarted","Data":"2c52a8ed3dc0cd33859f81b11125614ca3c3a005d200068df8b2b4b4305c7083"} Jul 24 19:32:15.423315 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:15.423262 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" podStartSLOduration=1.423245481 podStartE2EDuration="1.423245481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 19:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:32:15.421816898 +0000 UTC m=+2930.498329458" watchObservedRunningTime="2026-07-24 19:32:15.423245481 +0000 UTC m=+2930.499758038" Jul 24 19:32:16.413648 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:16.413614 2575 generic.go:356] "Generic (PLEG): container finished" podID="a4d8e53a-a447-4aad-80ae-2447e70dcbb7" containerID="0ea92e8a0712e6a3d9b8f7a4ac83d9813e4962f0942efecca24f65e83f37bb42" exitCode=0 Jul 24 19:32:16.413648 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:16.413651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" event={"ID":"a4d8e53a-a447-4aad-80ae-2447e70dcbb7","Type":"ContainerDied","Data":"0ea92e8a0712e6a3d9b8f7a4ac83d9813e4962f0942efecca24f65e83f37bb42"} Jul 24 19:32:17.568243 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:17.568216 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" Jul 24 19:32:17.642050 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:17.642016 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d8e53a-a447-4aad-80ae-2447e70dcbb7-maas-api-service-ca\") pod \"a4d8e53a-a447-4aad-80ae-2447e70dcbb7\" (UID: \"a4d8e53a-a447-4aad-80ae-2447e70dcbb7\") " Jul 24 19:32:17.642388 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:17.642363 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d8e53a-a447-4aad-80ae-2447e70dcbb7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4d8e53a-a447-4aad-80ae-2447e70dcbb7" (UID: "a4d8e53a-a447-4aad-80ae-2447e70dcbb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:32:17.742822 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:17.742712 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d8e53a-a447-4aad-80ae-2447e70dcbb7-maas-api-service-ca\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:32:18.429529 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:18.429497 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" Jul 24 19:32:18.429769 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:18.429533 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7" event={"ID":"a4d8e53a-a447-4aad-80ae-2447e70dcbb7","Type":"ContainerDied","Data":"2c52a8ed3dc0cd33859f81b11125614ca3c3a005d200068df8b2b4b4305c7083"} Jul 24 19:32:18.429769 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:18.429580 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c52a8ed3dc0cd33859f81b11125614ca3c3a005d200068df8b2b4b4305c7083" Jul 24 19:32:24.893586 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:24.893541 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276"] Jul 24 19:32:24.896402 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:24.896376 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-48f4d2-29748690-lx276"] Jul 24 19:32:24.920794 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:24.920761 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm"] Jul 24 19:32:24.921028 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:24.921003 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" podUID="383e19d6-bd28-4c53-9061-35e36a903243" containerName="payload-processing" containerID="cri-o://8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79" gracePeriod=30 Jul 24 19:32:25.176421 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.176387 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:32:25.205422 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.205329 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf5qg\" (UniqueName: \"kubernetes.io/projected/383e19d6-bd28-4c53-9061-35e36a903243-kube-api-access-kf5qg\") pod \"383e19d6-bd28-4c53-9061-35e36a903243\" (UID: \"383e19d6-bd28-4c53-9061-35e36a903243\") " Jul 24 19:32:25.205579 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.205477 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/383e19d6-bd28-4c53-9061-35e36a903243-plugins-config-volume\") pod \"383e19d6-bd28-4c53-9061-35e36a903243\" (UID: \"383e19d6-bd28-4c53-9061-35e36a903243\") " Jul 24 19:32:25.205867 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.205844 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383e19d6-bd28-4c53-9061-35e36a903243-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "383e19d6-bd28-4c53-9061-35e36a903243" (UID: "383e19d6-bd28-4c53-9061-35e36a903243"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 19:32:25.207506 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.207479 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383e19d6-bd28-4c53-9061-35e36a903243-kube-api-access-kf5qg" (OuterVolumeSpecName: "kube-api-access-kf5qg") pod "383e19d6-bd28-4c53-9061-35e36a903243" (UID: "383e19d6-bd28-4c53-9061-35e36a903243"). InnerVolumeSpecName "kube-api-access-kf5qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 19:32:25.307265 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.307225 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/383e19d6-bd28-4c53-9061-35e36a903243-plugins-config-volume\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:32:25.307265 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.307253 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf5qg\" (UniqueName: \"kubernetes.io/projected/383e19d6-bd28-4c53-9061-35e36a903243-kube-api-access-kf5qg\") on node \"ip-10-0-134-30.ec2.internal\" DevicePath \"\"" Jul 24 19:32:25.418078 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.417998 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90a2882c-af10-480a-8d4b-8e1c54575581" path="/var/lib/kubelet/pods/90a2882c-af10-480a-8d4b-8e1c54575581/volumes" Jul 24 19:32:25.479139 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.479101 2575 generic.go:356] "Generic (PLEG): container finished" podID="383e19d6-bd28-4c53-9061-35e36a903243" containerID="8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79" exitCode=0 Jul 24 19:32:25.479301 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.479155 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" Jul 24 19:32:25.479301 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.479180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" event={"ID":"383e19d6-bd28-4c53-9061-35e36a903243","Type":"ContainerDied","Data":"8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79"} Jul 24 19:32:25.479301 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.479259 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm" event={"ID":"383e19d6-bd28-4c53-9061-35e36a903243","Type":"ContainerDied","Data":"6ff5403758d89e908a9d2175c7f59b9f2bd6f55f44bf1af9d32a23c300153e7c"} Jul 24 19:32:25.479301 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.479287 2575 scope.go:117] "RemoveContainer" containerID="8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79" Jul 24 19:32:25.489167 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.489150 2575 scope.go:117] "RemoveContainer" containerID="8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79" Jul 24 19:32:25.489431 ip-10-0-134-30 kubenswrapper[2575]: E0724 19:32:25.489411 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79\": container with ID starting with 8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79 not found: ID does not exist" containerID="8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79" Jul 24 19:32:25.489504 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.489438 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79"} err="failed to get container status \"8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79\": rpc error: code = NotFound desc = could not find container \"8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79\": container with ID starting with 8c715cc5bd48ec69134e7e2aaab6dfc42bcc461d01169eba178d8cb7f5b9cb79 not found: ID does not exist" Jul 24 19:32:25.501297 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.501276 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm"] Jul 24 19:32:25.505414 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:25.505393 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-48f4d2-688ccc74f8-7w7wm"] Jul 24 19:32:27.417571 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:27.417539 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383e19d6-bd28-4c53-9061-35e36a903243" path="/var/lib/kubelet/pods/383e19d6-bd28-4c53-9061-35e36a903243/volumes" Jul 24 19:32:29.479077 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:29.479043 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7"] Jul 24 19:32:29.482181 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:29.482131 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-48f4d2-mqns7"] Jul 24 19:32:31.418380 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:31.418326 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d8e53a-a447-4aad-80ae-2447e70dcbb7" path="/var/lib/kubelet/pods/a4d8e53a-a447-4aad-80ae-2447e70dcbb7/volumes" Jul 24 19:32:41.941773 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:41.941729 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7gzl8_421e836f-94c8-469e-9af3-8b8732d6282d/manager/0.log" Jul 24 19:32:42.193805 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:42.193705 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54c89c5bc7-bpn4j_425ac1be-3355-4c91-8e9c-06e526b7e2e9/manager/0.log" Jul 24 19:32:42.300278 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:42.300251 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-cw2kq_a96b24b8-3be4-4d40-86ab-4041b8edd223/server/0.log" Jul 24 19:32:42.427564 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:42.427527 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-45v4h_238e37c5-ac35-446e-80c5-4fd5ba500e2a/manager/0.log" Jul 24 19:32:42.540117 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:42.540044 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6559d49d8b-fmds8_bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe/copy-manifests/0.log" Jul 24 19:32:42.546684 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:42.546661 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6559d49d8b-fmds8_bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe/manager/0.log" Jul 24 19:32:43.124673 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:43.124637 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f5d8786fb-4hrxs_23891aef-a3ee-4728-9f4d-5edb97010bec/maas-api/0.log" Jul 24 19:32:43.243114 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:43.243083 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748660-9tbbg_37c4967b-eda3-40b4-bd98-2c69d5582d4c/cleanup/0.log" Jul 24 19:32:43.349676 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:43.349650 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748675-6q7jk_13f45e1d-9790-475e-99af-f25dc3d9bab6/cleanup/0.log" Jul 24 19:32:43.460914 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:43.460885 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748690-tdx89_88b72634-10b9-4a12-a08e-270a054e04cf/cleanup/0.log" Jul 24 19:32:43.570324 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:43.570293 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-99zqr_bc85abbe-1231-4fe3-a47d-9aff88e08c22/postgres/0.log" Jul 24 19:32:44.940246 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:44.940221 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-c5q5c_dbf1c171-593d-4237-8c93-f068ee65eb8b/manager/0.log" Jul 24 19:32:45.046803 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:45.046776 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nmpjl_f4cddd24-220b-4224-ac3c-59328a709f56/manager/0.log" Jul 24 19:32:45.267778 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:45.267637 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-j8c9x_040b4576-d70e-461b-bb8b-315609527182/registry-server/0.log" Jul 24 19:32:45.382170 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:45.382131 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-lqz4k_4a98bc24-4210-4e6e-af0b-4b45b943c9c1/manager/0.log" Jul 24 19:32:45.609057 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:45.608944 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tlzpk_42836617-36e5-4fec-8cf8-5e50a3dd749b/manager/0.log" Jul 24 19:32:46.205761 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:46.205718 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d65fbd68-jmdst_e6e3085d-cade-4c35-9802-156cbcccaad5/kube-auth-proxy/0.log" Jul 24 19:32:46.548895 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:46.548788 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wjp2q_408c6a82-efa6-4c5d-99be-66f3fb7efdd7/payload-pre-processing/0.log" Jul 24 19:32:46.661956 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:46.661924 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vqv6d_be404140-206a-4cb9-b1e2-191c44c91c7f/payload-processing/0.log" Jul 24 19:32:46.774763 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:46.774714 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f4d68c8f7-vqjh4_ecfae2bd-f425-407e-b825-d5505cd538ef/router/0.log" Jul 24 19:32:47.108949 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:47.108920 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-gsc6h_f58b7a06-38bb-442a-940c-58a647c44d28/main/0.log" Jul 24 19:32:47.558976 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:47.558945 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-8p9qc_ef53f9fb-b3e8-4339-b2a6-3dfba7892130/main/0.log" Jul 24 19:32:55.500662 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:55.500633 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5q8wz_2a976a8d-b47d-4dbb-a8cd-f9a7170875a8/global-pull-secret-syncer/0.log" Jul 24 19:32:55.725696 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:55.725667 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rlnzb_fee3297a-eb1f-4807-b94b-a8b57a303292/konnectivity-agent/0.log" Jul 24 19:32:55.747545 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:32:55.747519 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-30.ec2.internal_2e352d90755072d947bfa04f22592e32/haproxy/0.log" Jul 24 19:33:00.502583 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:00.502542 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-c5q5c_dbf1c171-593d-4237-8c93-f068ee65eb8b/manager/0.log" Jul 24 19:33:00.546895 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:00.546851 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-nmpjl_f4cddd24-220b-4224-ac3c-59328a709f56/manager/0.log" Jul 24 19:33:00.626971 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:00.626938 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-j8c9x_040b4576-d70e-461b-bb8b-315609527182/registry-server/0.log" Jul 24 19:33:00.748670 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:00.748568 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-lqz4k_4a98bc24-4210-4e6e-af0b-4b45b943c9c1/manager/0.log" Jul 24 19:33:00.836314 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:00.836279 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tlzpk_42836617-36e5-4fec-8cf8-5e50a3dd749b/manager/0.log" Jul 24 19:33:02.829201 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:02.829164 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-qtf2r_9ca07cf2-39b7-4962-ad44-fb73e63c4005/cluster-monitoring-operator/0.log" Jul 24 19:33:03.104860 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:03.104728 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rwlb_8872ad15-0b96-43dc-96a3-16e84a86af39/node-exporter/0.log" Jul 24 19:33:03.146245 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:03.146223 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rwlb_8872ad15-0b96-43dc-96a3-16e84a86af39/kube-rbac-proxy/0.log" Jul 24 19:33:03.174511 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:03.174486 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8rwlb_8872ad15-0b96-43dc-96a3-16e84a86af39/init-textfile/0.log" Jul 24 19:33:04.008868 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:04.008835 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-fvt2v_1769c8bd-be29-4c80-b446-523184de4032/prometheus-operator-admission-webhook/0.log" Jul 24 19:33:05.607282 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:05.607248 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-6fjlk_2fbf38d4-922e-494d-ae19-415d297e9f14/networking-console-plugin/0.log" Jul 24 19:33:06.230996 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.230967 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/1.log" Jul 24 19:33:06.237042 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.237015 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-p2c9m_c683dae8-6f44-455b-bcd6-d81c0834ad39/console-operator/2.log" Jul 24 19:33:06.386435 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.386402 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn"] Jul 24 19:33:06.386878 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.386865 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d8e53a-a447-4aad-80ae-2447e70dcbb7" containerName="revoke-keys" Jul 24 19:33:06.386939 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.386880 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d8e53a-a447-4aad-80ae-2447e70dcbb7" containerName="revoke-keys" Jul 24 19:33:06.386939 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.386893 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383e19d6-bd28-4c53-9061-35e36a903243" containerName="payload-processing" Jul 24 19:33:06.386939 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.386898 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="383e19d6-bd28-4c53-9061-35e36a903243" containerName="payload-processing" Jul 24 19:33:06.387035 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.386979 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="383e19d6-bd28-4c53-9061-35e36a903243" containerName="payload-processing" Jul 24 19:33:06.387035 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.386989 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d8e53a-a447-4aad-80ae-2447e70dcbb7" containerName="revoke-keys" Jul 24 19:33:06.390128 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.390103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.392958 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.392931 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rq44j\"/\"openshift-service-ca.crt\"" Jul 24 19:33:06.393955 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.393934 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rq44j\"/\"kube-root-ca.crt\"" Jul 24 19:33:06.394061 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.393934 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rq44j\"/\"default-dockercfg-c7tk9\"" Jul 24 19:33:06.400321 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.400301 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn"] Jul 24 19:33:06.476851 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.476815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-lib-modules\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.476851 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.476849 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9sk\" (UniqueName: \"kubernetes.io/projected/f0414f04-2ee8-4532-827a-1cc29baf0e33-kube-api-access-pg9sk\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.477078 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.476925 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-proc\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.477078 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.477016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-podres\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.477078 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.477044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-sys\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.578151 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.578060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-proc\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.578151 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.578113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-podres\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.578151 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.578137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-sys\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.578416 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.578186 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-proc\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.578416 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.578205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-lib-modules\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.578416 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.578232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9sk\" (UniqueName: \"kubernetes.io/projected/f0414f04-2ee8-4532-827a-1cc29baf0e33-kube-api-access-pg9sk\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.578416 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.578247 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-podres\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.578416 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.578205 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-sys\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.578416 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.578285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f0414f04-2ee8-4532-827a-1cc29baf0e33-lib-modules\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.588440 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.588411 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9sk\" (UniqueName: \"kubernetes.io/projected/f0414f04-2ee8-4532-827a-1cc29baf0e33-kube-api-access-pg9sk\") pod \"perf-node-gather-daemonset-d2kvn\" (UID: \"f0414f04-2ee8-4532-827a-1cc29baf0e33\") " pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.701502 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.701470 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:06.791222 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.791194 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5f6bf56ff8-gnvh4_89d938c4-ee89-45f8-b063-0e421b7b64e4/console/0.log" Jul 24 19:33:06.830886 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.830811 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-xbdq4_88541e12-5e3b-47cd-947d-599b1501c178/download-server/0.log" Jul 24 19:33:06.841860 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:06.841839 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn"] Jul 24 19:33:06.843833 ip-10-0-134-30 kubenswrapper[2575]: W0724 19:33:06.843810 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf0414f04_2ee8_4532_827a_1cc29baf0e33.slice/crio-c2a394f3d23362915ae9d30190abec6de347f9d5873212de7b56a445b9347245 WatchSource:0}: Error finding container c2a394f3d23362915ae9d30190abec6de347f9d5873212de7b56a445b9347245: Status 404 returned error can't find the container with id c2a394f3d23362915ae9d30190abec6de347f9d5873212de7b56a445b9347245 Jul 24 19:33:07.431584 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:07.431554 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-bpcrc_9ca248b3-674e-4ee9-969a-7ad2f903b1a0/volume-data-source-validator/0.log" Jul 24 19:33:07.721408 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:07.721314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" event={"ID":"f0414f04-2ee8-4532-827a-1cc29baf0e33","Type":"ContainerStarted","Data":"4806d317ac431fb03f72ff242be81e7894980516d33b6808dde200ee07f6f405"} Jul 24 19:33:07.721802 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:07.721468 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:07.721802 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:07.721485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" event={"ID":"f0414f04-2ee8-4532-827a-1cc29baf0e33","Type":"ContainerStarted","Data":"c2a394f3d23362915ae9d30190abec6de347f9d5873212de7b56a445b9347245"} Jul 24 19:33:08.542965 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:08.542936 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-54h5x_0627a68a-b4bf-4eed-ae3d-6a3d4f97505e/dns/0.log" Jul 24 19:33:08.565193 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:08.565171 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-54h5x_0627a68a-b4bf-4eed-ae3d-6a3d4f97505e/kube-rbac-proxy/0.log" Jul 24 19:33:08.713879 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:08.713848 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pgx49_7d43cff0-e130-45e6-8fbc-e212008b30a4/dns-node-resolver/0.log" Jul 24 19:33:09.348994 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:09.348963 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s29nq_e6e2711c-af0a-41eb-b4d0-d1d86509fdc1/node-ca/0.log" Jul 24 19:33:10.610888 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:10.610859 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d65fbd68-jmdst_e6e3085d-cade-4c35-9802-156cbcccaad5/kube-auth-proxy/0.log" Jul 24 19:33:10.730511 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:10.730483 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wjp2q_408c6a82-efa6-4c5d-99be-66f3fb7efdd7/payload-pre-processing/0.log" Jul 24 19:33:10.758386 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:10.758363 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vqv6d_be404140-206a-4cb9-b1e2-191c44c91c7f/payload-processing/0.log" Jul 24 19:33:10.778942 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:10.778915 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f4d68c8f7-vqjh4_ecfae2bd-f425-407e-b825-d5505cd538ef/router/0.log" Jul 24 19:33:11.620530 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:11.620504 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5hsw_249a8d0d-04b2-4720-a52d-37c37ff7a4ac/serve-healthcheck-canary/0.log" Jul 24 19:33:12.141289 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:12.141262 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-6cl5p_5fc61054-0130-44ec-a896-11d6124b50d6/insights-operator/0.log" Jul 24 19:33:12.142095 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:12.142075 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-6cl5p_5fc61054-0130-44ec-a896-11d6124b50d6/insights-operator/1.log" Jul 24 19:33:12.165524 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:12.165506 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2ppz_85241c87-6a5d-4dce-bb06-9a46755ee37b/kube-rbac-proxy/0.log" Jul 24 19:33:12.188118 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:12.188094 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2ppz_85241c87-6a5d-4dce-bb06-9a46755ee37b/exporter/0.log" Jul 24 19:33:12.209844 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:12.209827 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c2ppz_85241c87-6a5d-4dce-bb06-9a46755ee37b/extractor/0.log" Jul 24 19:33:13.736998 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:13.736970 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" Jul 24 19:33:13.754816 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:13.754759 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rq44j/perf-node-gather-daemonset-d2kvn" podStartSLOduration=7.75472413 podStartE2EDuration="7.75472413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 19:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 19:33:07.759651835 +0000 UTC m=+2982.836164392" watchObservedRunningTime="2026-07-24 19:33:13.75472413 +0000 UTC m=+2988.831236698" Jul 24 19:33:14.504816 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:14.504792 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-7gzl8_421e836f-94c8-469e-9af3-8b8732d6282d/manager/0.log" Jul 24 19:33:14.999879 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:14.999848 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54c89c5bc7-bpn4j_425ac1be-3355-4c91-8e9c-06e526b7e2e9/manager/0.log" Jul 24 19:33:15.021324 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:15.021302 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-cw2kq_a96b24b8-3be4-4d40-86ab-4041b8edd223/server/0.log" Jul 24 19:33:15.100320 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:15.100294 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-45v4h_238e37c5-ac35-446e-80c5-4fd5ba500e2a/manager/0.log" Jul 24 19:33:15.127751 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:15.127702 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6559d49d8b-fmds8_bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe/manager/0.log" Jul 24 19:33:15.154876 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:15.154853 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6559d49d8b-fmds8_bcd4d9e6-ba36-478f-9171-7cffa5bd2dfe/copy-manifests/0.log" Jul 24 19:33:17.438815 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:17.438790 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d65fbd68-jmdst_e6e3085d-cade-4c35-9802-156cbcccaad5/kube-auth-proxy/0.log" Jul 24 19:33:17.538117 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:17.538086 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wjp2q_408c6a82-efa6-4c5d-99be-66f3fb7efdd7/payload-pre-processing/0.log" Jul 24 19:33:17.558239 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:17.558214 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vqv6d_be404140-206a-4cb9-b1e2-191c44c91c7f/payload-processing/0.log" Jul 24 19:33:17.575971 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:17.575952 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f4d68c8f7-vqjh4_ecfae2bd-f425-407e-b825-d5505cd538ef/router/0.log" Jul 24 19:33:19.257948 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:19.257915 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d65fbd68-jmdst_e6e3085d-cade-4c35-9802-156cbcccaad5/kube-auth-proxy/0.log" Jul 24 19:33:19.359488 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:19.359450 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wjp2q_408c6a82-efa6-4c5d-99be-66f3fb7efdd7/payload-pre-processing/0.log" Jul 24 19:33:19.380025 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:19.379998 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vqv6d_be404140-206a-4cb9-b1e2-191c44c91c7f/payload-processing/0.log" Jul 24 19:33:19.396319 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:19.396286 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f4d68c8f7-vqjh4_ecfae2bd-f425-407e-b825-d5505cd538ef/router/0.log" Jul 24 19:33:22.040021 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:22.039990 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-2fczv_4fe1e19b-8c9a-4915-8b6f-391712f0fed2/migrator/0.log" Jul 24 19:33:22.061378 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:22.061355 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-2fczv_4fe1e19b-8c9a-4915-8b6f-391712f0fed2/graceful-termination/0.log" Jul 24 19:33:22.448868 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:22.448831 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z_4abb5614-6d9a-42f6-a655-cfbc38b39382/kube-storage-version-migrator-operator/1.log" Jul 24 19:33:22.449759 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:22.449717 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-fwd4z_4abb5614-6d9a-42f6-a655-cfbc38b39382/kube-storage-version-migrator-operator/0.log" Jul 24 19:33:23.431898 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:23.431870 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psph6_14752100-7a25-4f88-abc5-c6903e11289e/kube-multus-additional-cni-plugins/0.log" Jul 24 19:33:23.453414 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:23.453387 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psph6_14752100-7a25-4f88-abc5-c6903e11289e/egress-router-binary-copy/0.log" Jul 24 19:33:23.475174 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:23.475155 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psph6_14752100-7a25-4f88-abc5-c6903e11289e/cni-plugins/0.log" Jul 24 19:33:23.499874 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:23.499851 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psph6_14752100-7a25-4f88-abc5-c6903e11289e/bond-cni-plugin/0.log" Jul 24 19:33:23.526410 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:23.526384 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psph6_14752100-7a25-4f88-abc5-c6903e11289e/routeoverride-cni/0.log" Jul 24 19:33:23.549127 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:23.549104 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psph6_14752100-7a25-4f88-abc5-c6903e11289e/whereabouts-cni-bincopy/0.log" Jul 24 19:33:23.570816 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:23.570791 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-psph6_14752100-7a25-4f88-abc5-c6903e11289e/whereabouts-cni/0.log" Jul 24 19:33:23.917152 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:23.917122 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bfzgn_d6d3da16-d2a0-4046-b297-759095781b40/kube-multus/0.log" Jul 24 19:33:24.027986 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:24.027960 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-khpvw_3ed67684-a72a-4ec0-9a45-3871b53a3e72/network-metrics-daemon/0.log" Jul 24 19:33:24.048419 ip-10-0-134-30 kubenswrapper[2575]: I0724 19:33:24.048393 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-khpvw_3ed67684-a72a-4ec0-9a45-3871b53a3e72/kube-rbac-proxy/0.log"