Jul 30 10:48:11.911341 ip-10-0-138-131 systemd[1]: Starting Kubernetes Kubelet... Jul 30 10:48:12.315136 ip-10-0-138-131 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 10:48:12.315136 ip-10-0-138-131 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 10:48:12.315136 ip-10-0-138-131 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 10:48:12.315136 ip-10-0-138-131 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 10:48:12.315136 ip-10-0-138-131 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 10:48:12.317002 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.316874 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 10:48:12.321142 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321126 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 10:48:12.321142 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321143 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321147 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321150 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321153 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321155 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321158 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321161 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321164 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321167 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321169 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321172 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321174 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321176 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321179 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321181 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321183 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321186 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321189 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321191 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 10:48:12.321204 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321193 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321196 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321198 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321200 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321202 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321205 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321207 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321210 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321213 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321215 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321218 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321220 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321223 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321225 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321227 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321229 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321232 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321234 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321236 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321238 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 10:48:12.321767 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321241 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321245 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321248 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321293 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321344 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321349 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321353 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321357 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321361 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321365 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321369 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321374 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321379 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321383 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321391 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321395 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321405 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321877 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321912 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 10:48:12.322466 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321919 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321924 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321930 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321935 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321939 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321944 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321948 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321953 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321957 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.321961 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322015 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322020 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322025 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322036 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322041 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322044 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322048 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322053 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322057 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322061 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 10:48:12.322973 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322064 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322069 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322077 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322099 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322103 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322107 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322110 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322114 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322118 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322121 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322125 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322130 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322135 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322142 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322151 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322155 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322159 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322163 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322168 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.322172 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 10:48:12.323450 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323112 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323122 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323126 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323129 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323133 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323136 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323139 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323142 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323144 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323147 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323149 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323152 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323154 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323157 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323159 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323162 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323164 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323166 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323170 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 10:48:12.323911 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323174 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323177 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323179 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323182 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323184 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323186 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323189 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323192 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323194 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323203 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323206 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323209 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323212 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323214 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323217 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323220 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323222 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323225 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323227 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 10:48:12.324384 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323229 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323232 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323234 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323236 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323239 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323241 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323244 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323246 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323248 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323251 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323253 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323255 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323258 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323260 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323262 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323265 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323267 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323269 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323271 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323274 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 10:48:12.324800 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323276 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323279 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323281 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323284 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323286 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323288 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323290 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323293 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323296 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323298 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323301 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323305 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323308 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323310 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323313 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323315 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323318 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323321 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323323 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323326 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 10:48:12.325271 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323329 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323332 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323334 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323336 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323338 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323340 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323343 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323345 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323347 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323350 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323352 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323354 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323356 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323359 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323361 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323363 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323365 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323367 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323370 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323373 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 10:48:12.325721 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323375 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323743 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323748 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323750 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323753 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323755 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323760 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323763 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323766 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323769 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323771 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323774 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323776 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323779 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323781 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323784 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323786 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323788 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323791 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 10:48:12.326189 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323793 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323796 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323798 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323800 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323803 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323805 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323807 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323810 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323813 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323815 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323817 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323820 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323823 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323826 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323828 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323831 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323833 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323836 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323838 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323840 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 10:48:12.326629 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323842 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323844 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323847 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323849 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323851 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323853 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323855 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323858 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323860 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323862 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323865 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323867 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323869 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323871 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323873 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323876 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323878 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323881 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323883 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323885 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 10:48:12.327103 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323887 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323890 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323892 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323895 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323898 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323901 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323903 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323905 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323908 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323910 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323914 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323917 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323920 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323923 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323925 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323928 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323930 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323933 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323935 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 10:48:12.327567 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323938 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323941 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323943 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323945 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323948 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323950 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323953 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323955 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323958 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323960 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323963 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323965 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323967 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323969 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323972 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323974 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323977 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323982 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323985 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 10:48:12.327987 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323987 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323989 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.323992 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324064 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324070 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324076 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324095 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324099 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324103 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324107 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324111 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324114 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324117 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324120 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324124 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324127 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324130 2575 flags.go:64] FLAG: --cgroup-root="" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324133 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324136 2575 flags.go:64] FLAG: --client-ca-file="" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324138 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324141 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324146 2575 flags.go:64] FLAG: --cluster-domain="" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324149 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324152 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 10:48:12.328425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324155 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324158 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324161 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324164 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324167 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324173 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324177 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324180 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324183 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324186 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324189 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324192 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324195 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324198 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324201 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324203 2575 flags.go:64] FLAG: --enable-server="true" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324206 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324211 2575 flags.go:64] FLAG: --event-burst="100" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324213 2575 flags.go:64] FLAG: --event-qps="50" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324216 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324219 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324221 2575 flags.go:64] FLAG: --eviction-hard="" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324225 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324228 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324230 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 10:48:12.328947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324233 2575 flags.go:64] FLAG: --eviction-soft="" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324236 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324238 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324241 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324244 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324246 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324249 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324252 2575 flags.go:64] FLAG: --feature-gates="" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324255 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324258 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324261 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324264 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324270 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324275 2575 flags.go:64] FLAG: --help="false" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324278 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324281 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324284 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324287 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324290 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324293 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324296 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324298 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324301 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 10:48:12.329540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324304 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324306 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324309 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324312 2575 flags.go:64] FLAG: --kube-reserved="" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324315 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324317 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324320 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324323 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324326 2575 flags.go:64] FLAG: --lock-file="" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324328 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324331 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324333 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324340 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324342 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324345 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324348 2575 flags.go:64] FLAG: --logging-format="text" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324350 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324354 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324356 2575 flags.go:64] FLAG: --manifest-url="" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324359 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324363 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324366 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324371 2575 flags.go:64] FLAG: --max-pods="110" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324375 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324378 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 10:48:12.330066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324381 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324383 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324387 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324389 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324392 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324400 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324403 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324406 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324409 2575 flags.go:64] FLAG: --pod-cidr="" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324411 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324416 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324418 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324422 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324424 2575 flags.go:64] FLAG: --port="10250" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324427 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324430 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03f7d01d4a485f4d6" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324432 2575 flags.go:64] FLAG: --qos-reserved="" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324435 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324438 2575 flags.go:64] FLAG: --register-node="true" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324442 2575 flags.go:64] FLAG: --register-with-taints="" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324445 2575 flags.go:64] FLAG: --registry-burst="10" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324448 2575 flags.go:64] FLAG: --registry-qps="5" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324450 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324453 2575 flags.go:64] FLAG: --reserved-memory="" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324456 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 10:48:12.330685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324459 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324462 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324464 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324467 2575 flags.go:64] FLAG: --runonce="false" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324469 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324473 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324477 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324479 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324482 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324485 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324487 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324490 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324493 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324495 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324498 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324501 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324504 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324506 2575 flags.go:64] FLAG: --system-cgroups="" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324509 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324514 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324516 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324519 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324523 2575 flags.go:64] FLAG: --tls-min-version="" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324526 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324528 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 10:48:12.331288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324531 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324535 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324537 2575 flags.go:64] FLAG: --v="2" Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324541 2575 flags.go:64] FLAG: --version="false" Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324545 2575 flags.go:64] FLAG: --vmodule="" Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324548 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.324551 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324655 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324659 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324662 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324665 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324668 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324672 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324675 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324678 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324681 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324684 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324687 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324690 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324693 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 10:48:12.331844 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324695 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324698 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324701 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324703 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324705 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324708 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324710 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324712 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324715 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324717 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324719 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324721 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324724 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324728 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324730 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324733 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324735 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324737 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324739 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324742 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 10:48:12.332323 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324744 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324746 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324750 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324753 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324757 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324760 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324763 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324766 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324768 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324771 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324773 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324776 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324778 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324780 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324783 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324785 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324787 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324789 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324792 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 10:48:12.332805 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324794 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324797 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324799 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324801 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324804 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324806 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324810 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324812 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324815 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324817 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324820 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324822 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324824 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324827 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324829 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324831 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324833 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324837 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324839 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324842 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 10:48:12.333328 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324844 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324846 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324849 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324851 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324853 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324856 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324858 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324860 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324864 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324867 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324870 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324872 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324874 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324876 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324879 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324881 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324883 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324886 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324889 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324891 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 10:48:12.333771 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324894 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324896 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324898 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324900 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324903 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324905 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.324908 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.325559 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.331727 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.331740 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.332470 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.332478 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332532 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332536 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332539 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 10:48:12.334238 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332543 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332545 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332548 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332551 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332555 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332558 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332560 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332563 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332565 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332568 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332571 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332574 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332576 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332578 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332581 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332583 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332586 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332588 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332590 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 10:48:12.334590 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332593 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332595 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332598 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332600 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332603 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332605 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332608 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332612 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332615 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332617 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332619 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332622 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332624 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332626 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332629 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332633 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332636 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332639 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332641 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332644 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 10:48:12.334998 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332646 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332648 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332651 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332653 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332656 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332658 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332660 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332662 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332665 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332668 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332671 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332674 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332677 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332679 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332682 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332684 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332686 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332689 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332691 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 10:48:12.335492 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332695 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332698 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332700 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332703 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332705 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332708 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332710 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332712 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332715 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332717 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332719 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332722 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332724 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332744 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332748 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332750 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332754 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332756 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332759 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332761 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 10:48:12.335916 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332764 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332766 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332769 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332771 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332773 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332775 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332778 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332780 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332783 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332785 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332787 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332790 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332792 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332795 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332798 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332800 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332802 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 10:48:12.336413 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332805 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.332810 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332911 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332915 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332917 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332920 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332923 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332925 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332929 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332932 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332935 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332939 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332941 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332944 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 10:48:12.336792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332946 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332949 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332951 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332953 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332956 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332958 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332961 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332963 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332965 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332967 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332969 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332971 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332975 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332977 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332980 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332982 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332985 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332987 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332989 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332992 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 10:48:12.337217 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332994 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332996 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.332998 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333001 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333010 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333012 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333014 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333017 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333019 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333021 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333023 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333026 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333028 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333031 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333033 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333036 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333039 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333042 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333045 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 10:48:12.337659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333047 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333050 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333053 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333055 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333058 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333060 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333063 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333066 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333068 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333070 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333073 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333076 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333083 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333099 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333103 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333106 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333109 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333111 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333114 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333116 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 10:48:12.338093 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333118 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333120 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333123 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333125 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333127 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333129 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333132 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333134 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333137 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333139 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333141 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333144 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333146 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333148 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333150 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333153 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333155 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333158 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333160 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333163 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 10:48:12.338544 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333165 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333168 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333170 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333173 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333176 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333178 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333180 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:12.333183 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.333187 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.333288 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.335057 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.336303 2575 server.go:1019] "Starting client certificate rotation" Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.336393 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 10:48:12.339009 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.336440 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 10:48:12.339332 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.338565 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 10:48:12.340986 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.340973 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 10:48:12.348604 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.348586 2575 log.go:25] "Validated CRI v1 runtime API" Jul 30 10:48:12.351859 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.351837 2575 log.go:25] "Validated CRI v1 image API" Jul 30 10:48:12.352597 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.352582 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 10:48:12.357814 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.357793 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ebf56f35-e625-4344-8f07-4821cbf78d74:/dev/nvme0n1p3 fa66902f-bc30-461e-8811-399c028153c4:/dev/nvme0n1p4] Jul 30 10:48:12.357859 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.357815 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 10:48:12.363551 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.363439 2575 manager.go:216] Machine: {Timestamp:2026-07-30 10:48:12.361703248 +0000 UTC m=+0.334564275 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100348 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2335e42d10335cc772ef459a415273 SystemUUID:ec2335e4-2d10-335c-c772-ef459a415273 BootID:769468b8-9dc6-4c9e-81d5-9f5d6b8f8b9a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1a:f1:17:ac:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1a:f1:17:ac:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:df:c9:67:ad:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 10:48:12.363551 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.363549 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 10:48:12.363656 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.363622 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 10:48:12.365823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.365809 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 10:48:12.366283 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.366261 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 10:48:12.366408 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.366288 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-131.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 10:48:12.367792 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.367782 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 10:48:12.367827 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.367794 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 10:48:12.367827 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.367816 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 10:48:12.368454 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.368444 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 10:48:12.369268 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.369260 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 10:48:12.370944 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.370934 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 30 10:48:12.371050 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.371042 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 10:48:12.371094 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.371070 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 30 10:48:12.371120 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.371094 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 10:48:12.371120 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.371111 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 10:48:12.371120 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.371120 2575 kubelet.go:398] "Adding apiserver pod source" Jul 30 10:48:12.371200 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.371128 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 10:48:12.371493 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.371482 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 10:48:12.371549 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.371499 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 10:48:12.373894 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.373879 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 10:48:12.375519 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.375506 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 10:48:12.375576 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.375535 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 10:48:12.377115 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377104 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 10:48:12.377168 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377120 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 10:48:12.377168 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377125 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 10:48:12.377168 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377130 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 10:48:12.377168 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377147 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 10:48:12.377168 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377157 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 10:48:12.377168 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377167 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 10:48:12.377352 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377175 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 10:48:12.377352 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377183 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 10:48:12.377352 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377191 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 10:48:12.377352 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377208 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 10:48:12.377352 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377217 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 10:48:12.377931 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377919 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 10:48:12.377965 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.377935 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 10:48:12.381233 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.381220 2575 server.go:1268] "Started kubelet" Jul 30 10:48:12.381401 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.381365 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 10:48:12.384218 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.383973 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 10:48:12.384218 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.384109 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 10:48:12.384858 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.384830 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 10:48:12.384858 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.384855 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-131.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 10:48:12.385201 ip-10-0-138-131 systemd[1]: Started Kubernetes Kubelet. Jul 30 10:48:12.387042 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.387022 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 30 10:48:12.387134 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.387072 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 10:48:12.389019 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.389000 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 10:48:12.389868 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.389850 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 10:48:12.391094 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.390192 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-131.ec2.internal.18c70b907758c862 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-131.ec2.internal,UID:ip-10-0-138-131.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-131.ec2.internal,},FirstTimestamp:2026-07-30 10:48:12.381194338 +0000 UTC m=+0.354055361,LastTimestamp:2026-07-30 10:48:12.381194338 +0000 UTC m=+0.354055361,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-131.ec2.internal,}" Jul 30 10:48:12.394654 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.394636 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 10:48:12.394654 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.394651 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 10:48:12.395078 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.394708 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:12.395228 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395192 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 10:48:12.395376 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395357 2575 factory.go:55] Registering systemd factory Jul 30 10:48:12.395376 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395377 2575 factory.go:223] Registration of the systemd container factory successfully Jul 30 10:48:12.395498 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395193 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 10:48:12.395498 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395418 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 10:48:12.395634 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395509 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 30 10:48:12.395634 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395518 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 30 10:48:12.395634 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395565 2575 factory.go:153] Registering CRI-O factory Jul 30 10:48:12.395634 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395580 2575 factory.go:223] Registration of the crio container factory successfully Jul 30 10:48:12.395634 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395627 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 10:48:12.395812 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395650 2575 factory.go:103] Registering Raw factory Jul 30 10:48:12.395812 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.395663 2575 manager.go:1196] Started watching for new ooms in manager Jul 30 10:48:12.396112 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.396077 2575 manager.go:349] Starting recovery of all containers Jul 30 10:48:12.397395 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.397370 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 30 10:48:12.397485 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.397435 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 30 10:48:12.404981 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.404963 2575 manager.go:354] Recovery completed Jul 30 10:48:12.408753 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.408742 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 10:48:12.410483 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.410456 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory" Jul 30 10:48:12.410540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.410487 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 10:48:12.410540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.410496 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID" Jul 30 10:48:12.411844 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.411802 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 10:48:12.411919 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.411845 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 10:48:12.411919 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.411863 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 30 10:48:12.411996 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.411964 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 10:48:12.412910 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.412884 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory" Jul 30 10:48:12.412980 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.412915 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 10:48:12.412980 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.412929 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID" Jul 30 10:48:12.413411 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.413284 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-131.ec2.internal.18c70b9079179212 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-131.ec2.internal,UID:ip-10-0-138-131.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-131.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-131.ec2.internal,},FirstTimestamp:2026-07-30 10:48:12.410475026 +0000 UTC m=+0.383336049,LastTimestamp:2026-07-30 10:48:12.410475026 +0000 UTC m=+0.383336049,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-131.ec2.internal,}" Jul 30 10:48:12.413744 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.413732 2575 policy_none.go:49] "None policy: Start" Jul 30 10:48:12.413798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.413752 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 10:48:12.413798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.413765 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 10:48:12.413998 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.413982 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gfcfl" Jul 30 10:48:12.414713 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.414702 2575 policy_none.go:47] "Start" Jul 30 10:48:12.423424 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.423242 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-131.ec2.internal.18c70b907917d1c1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-131.ec2.internal,UID:ip-10-0-138-131.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-131.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-131.ec2.internal,},FirstTimestamp:2026-07-30 10:48:12.410491329 +0000 UTC m=+0.383352352,LastTimestamp:2026-07-30 10:48:12.410491329 +0000 UTC m=+0.383352352,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-131.ec2.internal,}" Jul 30 10:48:12.426995 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.426979 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gfcfl" Jul 30 10:48:12.435117 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.435013 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-131.ec2.internal.18c70b907917f104 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-131.ec2.internal,UID:ip-10-0-138-131.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-131.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-131.ec2.internal,},FirstTimestamp:2026-07-30 10:48:12.410499332 +0000 UTC m=+0.383360355,LastTimestamp:2026-07-30 10:48:12.410499332 +0000 UTC m=+0.383360355,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-131.ec2.internal,}" Jul 30 10:48:12.445716 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.445698 2575 manager.go:337] "Starting Device Plugin manager" Jul 30 10:48:12.445840 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.445739 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 10:48:12.445840 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.445749 2575 server.go:87] "Starting device plugin registration server" Jul 30 10:48:12.446058 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.446045 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 10:48:12.446138 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.446061 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 10:48:12.446194 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.446179 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 10:48:12.446278 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.446266 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 10:48:12.446278 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.446279 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 10:48:12.446701 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.446679 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 10:48:12.446769 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.446730 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:12.519374 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.519297 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 10:48:12.521191 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.520433 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 10:48:12.521191 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.520464 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 10:48:12.521191 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.520483 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 10:48:12.521191 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.520515 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 30 10:48:12.523397 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.523378 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 10:48:12.546555 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.546532 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 10:48:12.547511 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.547496 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory" Jul 30 10:48:12.547599 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.547523 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 10:48:12.547599 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.547533 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID" Jul 30 10:48:12.547599 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.547553 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.558597 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.558577 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.558646 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.558601 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-131.ec2.internal\": node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:12.588313 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.588293 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:12.621158 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.621129 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal"] Jul 30 10:48:12.621297 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.621195 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 10:48:12.622068 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.622053 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory" Jul 30 10:48:12.622172 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.622083 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 10:48:12.622172 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.622107 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID" Jul 30 10:48:12.624105 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.624074 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 10:48:12.624182 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.624169 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.624240 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.624203 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 10:48:12.624835 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.624819 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory" Jul 30 10:48:12.624932 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.624825 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory" Jul 30 10:48:12.624932 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.624866 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 10:48:12.624932 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.624875 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID" Jul 30 10:48:12.624932 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.624847 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 10:48:12.624932 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.624905 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID" Jul 30 10:48:12.626728 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.626716 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.626771 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.626743 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 10:48:12.627440 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.627422 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory" Jul 30 10:48:12.627512 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.627446 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 10:48:12.627512 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.627455 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID" Jul 30 10:48:12.653812 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.653789 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-131.ec2.internal\" not found" node="ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.657899 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.657881 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-131.ec2.internal\" not found" node="ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.688520 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.688498 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:12.696741 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.696720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/675b44794ae9b75b8a3551e08f825170-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"675b44794ae9b75b8a3551e08f825170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.696830 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.696745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/675b44794ae9b75b8a3551e08f825170-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"675b44794ae9b75b8a3551e08f825170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.696830 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.696762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e190c0b1e0328f4dc4c1440290f7e5c1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-131.ec2.internal\" (UID: \"e190c0b1e0328f4dc4c1440290f7e5c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.788924 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.788887 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:12.797214 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.797188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/675b44794ae9b75b8a3551e08f825170-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"675b44794ae9b75b8a3551e08f825170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.797333 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.797217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/675b44794ae9b75b8a3551e08f825170-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"675b44794ae9b75b8a3551e08f825170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.797333 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.797244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e190c0b1e0328f4dc4c1440290f7e5c1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-131.ec2.internal\" (UID: \"e190c0b1e0328f4dc4c1440290f7e5c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.797333 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.797286 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/675b44794ae9b75b8a3551e08f825170-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"675b44794ae9b75b8a3551e08f825170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.797333 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.797285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e190c0b1e0328f4dc4c1440290f7e5c1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-131.ec2.internal\" (UID: \"e190c0b1e0328f4dc4c1440290f7e5c1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.797333 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.797286 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/675b44794ae9b75b8a3551e08f825170-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal\" (UID: \"675b44794ae9b75b8a3551e08f825170\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.889584 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.889538 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:12.956925 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.956891 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.961679 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:12.961536 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" Jul 30 10:48:12.990071 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:12.990041 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:13.090531 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:13.090452 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:13.190868 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:13.190828 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:13.231705 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.231678 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 10:48:13.291374 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:13.291341 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:13.335709 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.335685 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 10:48:13.336182 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.335831 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 10:48:13.336182 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.335852 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 10:48:13.392112 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:13.392016 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:13.434644 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.434602 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 10:43:12 +0000 UTC" deadline="2028-01-22 00:16:54.360216284 +0000 UTC" Jul 30 10:48:13.434644 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.434635 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12973h28m40.925584618s" Jul 30 10:48:13.472291 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.472270 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 10:48:13.489698 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.489666 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 10:48:13.491024 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.491005 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientMemory" Jul 30 10:48:13.491172 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.491033 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 10:48:13.491172 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.491047 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeHasSufficientPID" Jul 30 10:48:13.492095 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:13.492058 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-131.ec2.internal\" not found" Jul 30 10:48:13.537281 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.537251 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 10:48:13.590921 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:13.590893 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675b44794ae9b75b8a3551e08f825170.slice/crio-66029cfa802f86381fc3ada86cb9baa7c6a33bd2895e4cf733f9a6e3b9f2ab32 WatchSource:0}: Error finding container 66029cfa802f86381fc3ada86cb9baa7c6a33bd2895e4cf733f9a6e3b9f2ab32: Status 404 returned error can't find the container with id 66029cfa802f86381fc3ada86cb9baa7c6a33bd2895e4cf733f9a6e3b9f2ab32 Jul 30 10:48:13.591316 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:13.591300 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode190c0b1e0328f4dc4c1440290f7e5c1.slice/crio-703ad3cfa7ea192abb405965d61e821c7af70d60a34a2b5fd8c2a61c6df7d3d1 WatchSource:0}: Error finding container 703ad3cfa7ea192abb405965d61e821c7af70d60a34a2b5fd8c2a61c6df7d3d1: Status 404 returned error can't find the container with id 703ad3cfa7ea192abb405965d61e821c7af70d60a34a2b5fd8c2a61c6df7d3d1 Jul 30 10:48:13.594784 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.594766 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" Jul 30 10:48:13.595940 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.595925 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 10:48:13.608400 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.608385 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 10:48:13.609390 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.609378 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" Jul 30 10:48:13.616824 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:13.616811 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 10:48:14.224168 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.224140 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 10:48:14.372997 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.372969 2575 apiserver.go:52] "Watching apiserver" Jul 30 10:48:14.377782 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.377758 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 10:48:14.378913 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.378889 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-452jf","kube-system/konnectivity-agent-dx77j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm","openshift-dns/node-resolver-f5vrl","openshift-image-registry/node-ca-qwpkd","openshift-multus/multus-8kwj2","openshift-multus/multus-additional-cni-plugins-dspfd","openshift-multus/network-metrics-daemon-chjhd","kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal","openshift-cluster-node-tuning-operator/tuned-nzkqn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal","openshift-network-diagnostics/network-check-target-cdb5g","openshift-network-operator/iptables-alerter-b8skc","openshift-ovn-kubernetes/ovnkube-node-2z7n2"] Jul 30 10:48:14.381331 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.381307 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.383256 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.383229 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 10:48:14.383352 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.383294 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m7j5j\"" Jul 30 10:48:14.383436 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.383413 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.383663 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.383578 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 10:48:14.383663 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.383421 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 10:48:14.383663 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.383628 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 10:48:14.384067 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.383860 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 10:48:14.385389 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.385171 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pb95p\"" Jul 30 10:48:14.385389 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.385193 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 10:48:14.385534 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.385390 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 10:48:14.385534 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.385458 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 10:48:14.385626 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.385609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.387032 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.387015 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pg7ns\"" Jul 30 10:48:14.387134 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.387014 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 10:48:14.387193 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.387134 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 10:48:14.387193 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.387182 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 10:48:14.387857 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.387836 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.390010 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.389953 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7xsrz\"" Jul 30 10:48:14.390926 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.390478 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 10:48:14.390926 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.390753 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 10:48:14.391060 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.390949 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.392749 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.392731 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 10:48:14.392826 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.392802 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 10:48:14.392902 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.392850 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zkcqp\"" Jul 30 10:48:14.392963 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.392926 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 10:48:14.393127 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.393109 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.394731 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.394712 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 10:48:14.394731 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.394727 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rgfwk\"" Jul 30 10:48:14.395406 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.395176 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.395964 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.395687 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 10:48:14.397233 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.397214 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 10:48:14.397346 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.397304 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 10:48:14.397399 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.397352 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dhsm7\"" Jul 30 10:48:14.397399 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.397366 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 10:48:14.397399 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.397373 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 10:48:14.397887 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.397702 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 10:48:14.398120 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.398103 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 10:48:14.398199 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.398161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:14.398254 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.398224 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:14.400206 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.400054 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:14.402607 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.402588 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 10:48:14.402699 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.402630 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 10:48:14.402699 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.402637 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-b64dz\"" Jul 30 10:48:14.403769 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-systemd-units\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.403847 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403781 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.403847 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403806 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-socket-dir-parent\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.403847 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-run-k8s-cni-cncf-io\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.403988 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgx5\" (UniqueName: \"kubernetes.io/projected/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-kube-api-access-kdgx5\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.403988 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-cni-netd\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.403988 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-kubelet\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.403988 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-var-lib-openvswitch\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.403988 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-etc-openvswitch\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.403988 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403984 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.404272 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.403995 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-run-ovn-kubernetes\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.404272 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404053 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffc50509-0334-4ee6-b2c8-ae756587b675-ovnkube-config\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.404272 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.404272 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q68w\" (UniqueName: \"kubernetes.io/projected/f8c00f42-bdbc-49fe-9c83-e186f54e6ac8-kube-api-access-7q68w\") pod \"iptables-alerter-b8skc\" (UID: \"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8\") " pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.404272 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fdjt\" (UniqueName: \"kubernetes.io/projected/177e8448-e3d4-40b5-aa14-9889d7901b64-kube-api-access-2fdjt\") pod \"node-ca-qwpkd\" (UID: \"177e8448-e3d4-40b5-aa14-9889d7901b64\") " pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.404272 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-daemon-config\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.404272 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:14.404272 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffc50509-0334-4ee6-b2c8-ae756587b675-ovnkube-script-lib\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404279 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dthlr\" (UniqueName: \"kubernetes.io/projected/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-kube-api-access-dthlr\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-cni-dir\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404339 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-run-netns\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404362 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-system-cni-dir\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-socket-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8c00f42-bdbc-49fe-9c83-e186f54e6ac8-host-slash\") pod \"iptables-alerter-b8skc\" (UID: \"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8\") " pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404445 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-var-lib-cni-multus\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffc50509-0334-4ee6-b2c8-ae756587b675-ovn-node-metrics-cert\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msxrz\" (UniqueName: \"kubernetes.io/projected/ffc50509-0334-4ee6-b2c8-ae756587b675-kube-api-access-msxrz\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-var-lib-cni-bin\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404547 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-conf-dir\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.404611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8307c4d-a9aa-4e53-9b75-ca7bc4080028-hosts-file\") pod \"node-resolver-f5vrl\" (UID: \"f8307c4d-a9aa-4e53-9b75-ca7bc4080028\") " pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7m2l\" (UniqueName: \"kubernetes.io/projected/f8307c4d-a9aa-4e53-9b75-ca7bc4080028-kube-api-access-r7m2l\") pod \"node-resolver-f5vrl\" (UID: \"f8307c4d-a9aa-4e53-9b75-ca7bc4080028\") " pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-cni-bin\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffc50509-0334-4ee6-b2c8-ae756587b675-env-overrides\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-cnibin\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404714 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404735 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-sys-fs\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-var-lib-kubelet\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-node-log\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404793 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-device-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-os-release\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-hostroot\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404885 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-run-systemd\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-log-socket\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.405144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-registration-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.404988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-cnibin\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405008 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-slash\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405029 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-os-release\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-system-cni-dir\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405075 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-run-openvswitch\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405110 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nww\" (UniqueName: \"kubernetes.io/projected/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-kube-api-access-h8nww\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-run-multus-certs\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405198 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f31dbccc-5060-4cb2-9df5-279124fcd806-dbus\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-run-netns\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8c00f42-bdbc-49fe-9c83-e186f54e6ac8-iptables-alerter-script\") pod \"iptables-alerter-b8skc\" (UID: \"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8\") " pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-cni-binary-copy\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-etc-kubernetes\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-run-ovn\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.405867 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/177e8448-e3d4-40b5-aa14-9889d7901b64-host\") pod \"node-ca-qwpkd\" (UID: \"177e8448-e3d4-40b5-aa14-9889d7901b64\") " pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.406568 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/177e8448-e3d4-40b5-aa14-9889d7901b64-serviceca\") pod \"node-ca-qwpkd\" (UID: \"177e8448-e3d4-40b5-aa14-9889d7901b64\") " pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.406568 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405442 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f31dbccc-5060-4cb2-9df5-279124fcd806-kubelet-config\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:14.406568 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8307c4d-a9aa-4e53-9b75-ca7bc4080028-tmp-dir\") pod \"node-resolver-f5vrl\" (UID: \"f8307c4d-a9aa-4e53-9b75-ca7bc4080028\") " pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.406568 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.405783 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:14.406568 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.405840 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:14.407130 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.407113 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 10:48:14.407259 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.407241 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 10:48:14.407326 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.407311 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 10:48:14.407460 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.407441 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-shpkg\"" Jul 30 10:48:14.408196 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.408180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:14.408276 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.408251 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:14.429322 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.429299 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xfs4c" Jul 30 10:48:14.445204 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.445050 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xfs4c" Jul 30 10:48:14.496560 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.496486 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 10:48:14.506024 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.505988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8pc\" (UniqueName: \"kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc\") pod \"network-check-target-cdb5g\" (UID: \"d86e960e-cae8-4510-afcd-37dc31b105e6\") " pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:14.506195 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-socket-dir-parent\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.506195 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-run-k8s-cni-cncf-io\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.506195 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgx5\" (UniqueName: \"kubernetes.io/projected/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-kube-api-access-kdgx5\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.506195 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-cni-netd\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.506195 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-sysctl-d\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.506195 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-kubelet\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.506195 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-var-lib-openvswitch\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-etc-openvswitch\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-run-ovn-kubernetes\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506260 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffc50509-0334-4ee6-b2c8-ae756587b675-ovnkube-config\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q68w\" (UniqueName: \"kubernetes.io/projected/f8c00f42-bdbc-49fe-9c83-e186f54e6ac8-kube-api-access-7q68w\") pod \"iptables-alerter-b8skc\" (UID: \"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8\") " pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fdjt\" (UniqueName: \"kubernetes.io/projected/177e8448-e3d4-40b5-aa14-9889d7901b64-kube-api-access-2fdjt\") pod \"node-ca-qwpkd\" (UID: \"177e8448-e3d4-40b5-aa14-9889d7901b64\") " pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506367 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-kubernetes\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-tuned\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-daemon-config\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506412 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-kubelet\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:14.506524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffc50509-0334-4ee6-b2c8-ae756587b675-ovnkube-script-lib\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dthlr\" (UniqueName: \"kubernetes.io/projected/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-kube-api-access-dthlr\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506563 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-run-ovn-kubernetes\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506584 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-cni-dir\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-run-netns\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506632 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-var-lib-openvswitch\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-system-cni-dir\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506658 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-socket-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-etc-openvswitch\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8c00f42-bdbc-49fe-9c83-e186f54e6ac8-host-slash\") pod \"iptables-alerter-b8skc\" (UID: \"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8\") " pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-var-lib-cni-multus\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffc50509-0334-4ee6-b2c8-ae756587b675-ovn-node-metrics-cert\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.506809 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxrz\" (UniqueName: \"kubernetes.io/projected/ffc50509-0334-4ee6-b2c8-ae756587b675-kube-api-access-msxrz\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-var-lib-cni-bin\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-conf-dir\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8307c4d-a9aa-4e53-9b75-ca7bc4080028-hosts-file\") pod \"node-resolver-f5vrl\" (UID: \"f8307c4d-a9aa-4e53-9b75-ca7bc4080028\") " pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.506991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7m2l\" (UniqueName: \"kubernetes.io/projected/f8307c4d-a9aa-4e53-9b75-ca7bc4080028-kube-api-access-r7m2l\") pod \"node-resolver-f5vrl\" (UID: \"f8307c4d-a9aa-4e53-9b75-ca7bc4080028\") " pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.507056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-cni-bin\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffc50509-0334-4ee6-b2c8-ae756587b675-env-overrides\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-cnibin\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-sys-fs\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-run\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-var-lib-kubelet\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507199 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2z5m\" (UniqueName: \"kubernetes.io/projected/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-kube-api-access-z2z5m\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-var-lib-kubelet\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-node-log\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507300 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ffc50509-0334-4ee6-b2c8-ae756587b675-ovnkube-config\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507301 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ffc50509-0334-4ee6-b2c8-ae756587b675-ovnkube-script-lib\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-device-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-sys\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.507621 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret podName:f31dbccc-5060-4cb2-9df5-279124fcd806 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:15.007598484 +0000 UTC m=+2.980459500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret") pod "global-pull-secret-syncer-452jf" (UID: "f31dbccc-5060-4cb2-9df5-279124fcd806") : object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:14.507838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-var-lib-cni-multus\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507669 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-var-lib-cni-bin\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v549j\" (UniqueName: \"kubernetes.io/projected/3e7bd26f-41bc-4fda-8adc-f485de7344bb-kube-api-access-v549j\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-os-release\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-hostroot\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-run-systemd\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-log-socket\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507829 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-system-cni-dir\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-registration-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-modprobe-d\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507893 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-host\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-socket-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-cnibin\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-slash\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507937 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-os-release\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507980 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-run-k8s-cni-cncf-io\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.508569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e7bd26f-41bc-4fda-8adc-f485de7344bb-tmp\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-system-cni-dir\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-run-openvswitch\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-socket-dir-parent\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508121 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nww\" (UniqueName: \"kubernetes.io/projected/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-kube-api-access-h8nww\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-sysconfig\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508187 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-systemd\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd19302d-83a0-404d-923b-0ed52e8bcf51-agent-certs\") pod \"konnectivity-agent-dx77j\" (UID: \"dd19302d-83a0-404d-923b-0ed52e8bcf51\") " pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd19302d-83a0-404d-923b-0ed52e8bcf51-konnectivity-ca\") pod \"konnectivity-agent-dx77j\" (UID: \"dd19302d-83a0-404d-923b-0ed52e8bcf51\") " pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-run-netns\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-run-multus-certs\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f31dbccc-5060-4cb2-9df5-279124fcd806-dbus\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508578 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-run-netns\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508588 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-conf-dir\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-sysctl-conf\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.509380 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508624 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-sys-fs\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-lib-modules\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508658 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8307c4d-a9aa-4e53-9b75-ca7bc4080028-hosts-file\") pod \"node-resolver-f5vrl\" (UID: \"f8307c4d-a9aa-4e53-9b75-ca7bc4080028\") " pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8c00f42-bdbc-49fe-9c83-e186f54e6ac8-iptables-alerter-script\") pod \"iptables-alerter-b8skc\" (UID: \"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8\") " pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-cni-binary-copy\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-etc-kubernetes\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-run-ovn\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/177e8448-e3d4-40b5-aa14-9889d7901b64-host\") pod \"node-ca-qwpkd\" (UID: \"177e8448-e3d4-40b5-aa14-9889d7901b64\") " pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/177e8448-e3d4-40b5-aa14-9889d7901b64-serviceca\") pod \"node-ca-qwpkd\" (UID: \"177e8448-e3d4-40b5-aa14-9889d7901b64\") " pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508861 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-system-cni-dir\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f31dbccc-5060-4cb2-9df5-279124fcd806-kubelet-config\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8307c4d-a9aa-4e53-9b75-ca7bc4080028-tmp-dir\") pod \"node-resolver-f5vrl\" (UID: \"f8307c4d-a9aa-4e53-9b75-ca7bc4080028\") " pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.508974 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-systemd-units\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509023 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509532 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-cni-binary-copy\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.510183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-daemon-config\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509594 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-run-openvswitch\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509622 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-cni-netd\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509864 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-multus-cni-dir\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509917 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-os-release\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509945 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-hostroot\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509972 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-run-systemd\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.509998 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-log-socket\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510109 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-etc-selinux\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.507947 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f8c00f42-bdbc-49fe-9c83-e186f54e6ac8-host-slash\") pod \"iptables-alerter-b8skc\" (UID: \"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8\") " pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510248 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-registration-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510306 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510329 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-cnibin\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510348 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-slash\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510376 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510397 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-os-release\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-cnibin\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.510897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510432 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-node-log\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510463 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-device-dir\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510464 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-cni-bin\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510490 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-var-lib-kubelet\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510501 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/177e8448-e3d4-40b5-aa14-9889d7901b64-host\") pod \"node-ca-qwpkd\" (UID: \"177e8448-e3d4-40b5-aa14-9889d7901b64\") " pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510575 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-host-run-multus-certs\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510643 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-run-netns\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510694 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f31dbccc-5060-4cb2-9df5-279124fcd806-dbus\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510734 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510797 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ffc50509-0334-4ee6-b2c8-ae756587b675-env-overrides\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-etc-kubernetes\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510874 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-run-ovn\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510951 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f8c00f42-bdbc-49fe-9c83-e186f54e6ac8-iptables-alerter-script\") pod \"iptables-alerter-b8skc\" (UID: \"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8\") " pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.510991 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ffc50509-0334-4ee6-b2c8-ae756587b675-systemd-units\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.511038 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f31dbccc-5060-4cb2-9df5-279124fcd806-kubelet-config\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.511109 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8307c4d-a9aa-4e53-9b75-ca7bc4080028-tmp-dir\") pod \"node-resolver-f5vrl\" (UID: \"f8307c4d-a9aa-4e53-9b75-ca7bc4080028\") " pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.511385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ffc50509-0334-4ee6-b2c8-ae756587b675-ovn-node-metrics-cert\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.511390 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/177e8448-e3d4-40b5-aa14-9889d7901b64-serviceca\") pod \"node-ca-qwpkd\" (UID: \"177e8448-e3d4-40b5-aa14-9889d7901b64\") " pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.511747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.511686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-cni-binary-copy\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.526040 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.525965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" event={"ID":"e190c0b1e0328f4dc4c1440290f7e5c1","Type":"ContainerStarted","Data":"703ad3cfa7ea192abb405965d61e821c7af70d60a34a2b5fd8c2a61c6df7d3d1"} Jul 30 10:48:14.527871 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.527839 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" event={"ID":"675b44794ae9b75b8a3551e08f825170","Type":"ContainerStarted","Data":"66029cfa802f86381fc3ada86cb9baa7c6a33bd2895e4cf733f9a6e3b9f2ab32"} Jul 30 10:48:14.540141 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.539830 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxrz\" (UniqueName: \"kubernetes.io/projected/ffc50509-0334-4ee6-b2c8-ae756587b675-kube-api-access-msxrz\") pod \"ovnkube-node-2z7n2\" (UID: \"ffc50509-0334-4ee6-b2c8-ae756587b675\") " pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.540141 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.540070 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nww\" (UniqueName: \"kubernetes.io/projected/125f3a8d-6f3a-434c-b8d7-45d9f56b32e5-kube-api-access-h8nww\") pod \"multus-additional-cni-plugins-dspfd\" (UID: \"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5\") " pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.543681 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.543576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7m2l\" (UniqueName: \"kubernetes.io/projected/f8307c4d-a9aa-4e53-9b75-ca7bc4080028-kube-api-access-r7m2l\") pod \"node-resolver-f5vrl\" (UID: \"f8307c4d-a9aa-4e53-9b75-ca7bc4080028\") " pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.544242 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.544200 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fdjt\" (UniqueName: \"kubernetes.io/projected/177e8448-e3d4-40b5-aa14-9889d7901b64-kube-api-access-2fdjt\") pod \"node-ca-qwpkd\" (UID: \"177e8448-e3d4-40b5-aa14-9889d7901b64\") " pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.544342 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.544319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgx5\" (UniqueName: \"kubernetes.io/projected/ca678be3-1a7c-41dc-adb3-7fc44a1887ed-kube-api-access-kdgx5\") pod \"multus-8kwj2\" (UID: \"ca678be3-1a7c-41dc-adb3-7fc44a1887ed\") " pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.545469 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.545445 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q68w\" (UniqueName: \"kubernetes.io/projected/f8c00f42-bdbc-49fe-9c83-e186f54e6ac8-kube-api-access-7q68w\") pod \"iptables-alerter-b8skc\" (UID: \"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8\") " pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.548361 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.548335 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dthlr\" (UniqueName: \"kubernetes.io/projected/c95bb37e-4aed-48e4-bbf6-b152ca7b685d-kube-api-access-dthlr\") pod \"aws-ebs-csi-driver-node-m8jdm\" (UID: \"c95bb37e-4aed-48e4-bbf6-b152ca7b685d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.610277 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-run\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610285 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-var-lib-kubelet\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2z5m\" (UniqueName: \"kubernetes.io/projected/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-kube-api-access-z2z5m\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610335 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-sys\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v549j\" (UniqueName: \"kubernetes.io/projected/3e7bd26f-41bc-4fda-8adc-f485de7344bb-kube-api-access-v549j\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-modprobe-d\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-host\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e7bd26f-41bc-4fda-8adc-f485de7344bb-tmp\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610450 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-sysconfig\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-systemd\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd19302d-83a0-404d-923b-0ed52e8bcf51-agent-certs\") pod \"konnectivity-agent-dx77j\" (UID: \"dd19302d-83a0-404d-923b-0ed52e8bcf51\") " pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:14.610522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610519 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd19302d-83a0-404d-923b-0ed52e8bcf51-konnectivity-ca\") pod \"konnectivity-agent-dx77j\" (UID: \"dd19302d-83a0-404d-923b-0ed52e8bcf51\") " pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:14.611113 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-sysctl-conf\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.611113 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-lib-modules\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.611113 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610584 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:14.611113 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8pc\" (UniqueName: \"kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc\") pod \"network-check-target-cdb5g\" (UID: \"d86e960e-cae8-4510-afcd-37dc31b105e6\") " pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:14.611113 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-sysctl-d\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.611113 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-kubernetes\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.611113 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-tuned\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.611113 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610825 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-run\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.611113 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.610877 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-var-lib-kubelet\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.611520 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.611179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-sys\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.611520 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.611467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-modprobe-d\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.611520 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.611515 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-host\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.612552 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.611765 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-lib-modules\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.612552 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.611861 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-systemd\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.612552 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.611859 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-sysctl-d\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.612552 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.611935 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:14.612552 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.612027 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs podName:d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb nodeName:}" failed. No retries permitted until 2026-07-30 10:48:15.111980095 +0000 UTC m=+3.084841113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs") pod "network-metrics-daemon-chjhd" (UID: "d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:14.612552 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.612434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-kubernetes\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.612552 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.612446 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-sysctl-conf\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.612552 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.612495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-sysconfig\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.613118 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.612711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd19302d-83a0-404d-923b-0ed52e8bcf51-konnectivity-ca\") pod \"konnectivity-agent-dx77j\" (UID: \"dd19302d-83a0-404d-923b-0ed52e8bcf51\") " pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:14.613287 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.613268 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e7bd26f-41bc-4fda-8adc-f485de7344bb-etc-tuned\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.614791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.614772 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e7bd26f-41bc-4fda-8adc-f485de7344bb-tmp\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.615165 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.615149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd19302d-83a0-404d-923b-0ed52e8bcf51-agent-certs\") pod \"konnectivity-agent-dx77j\" (UID: \"dd19302d-83a0-404d-923b-0ed52e8bcf51\") " pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:14.621805 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.621784 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 10:48:14.621900 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.621810 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 10:48:14.621900 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.621823 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hb8pc for pod openshift-network-diagnostics/network-check-target-cdb5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:14.621900 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:14.621881 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc podName:d86e960e-cae8-4510-afcd-37dc31b105e6 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:15.121863412 +0000 UTC m=+3.094724443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hb8pc" (UniqueName: "kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc") pod "network-check-target-cdb5g" (UID: "d86e960e-cae8-4510-afcd-37dc31b105e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:14.623190 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.623153 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2z5m\" (UniqueName: \"kubernetes.io/projected/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-kube-api-access-z2z5m\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:14.628328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.628310 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v549j\" (UniqueName: \"kubernetes.io/projected/3e7bd26f-41bc-4fda-8adc-f485de7344bb-kube-api-access-v549j\") pod \"tuned-nzkqn\" (UID: \"3e7bd26f-41bc-4fda-8adc-f485de7344bb\") " pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:14.692437 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.692394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dspfd" Jul 30 10:48:14.701281 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.701260 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b8skc" Jul 30 10:48:14.711491 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.711465 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8kwj2" Jul 30 10:48:14.717700 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.717679 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" Jul 30 10:48:14.723172 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.723154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f5vrl" Jul 30 10:48:14.731636 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.731614 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qwpkd" Jul 30 10:48:14.737196 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.737176 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:14.750813 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.750754 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:14.756335 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:14.756311 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" Jul 30 10:48:15.012448 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.012363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:15.012605 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:15.012489 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:15.012605 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:15.012554 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret podName:f31dbccc-5060-4cb2-9df5-279124fcd806 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:16.012532577 +0000 UTC m=+3.985393591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret") pod "global-pull-secret-syncer-452jf" (UID: "f31dbccc-5060-4cb2-9df5-279124fcd806") : object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:15.113151 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.113115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:15.113300 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:15.113251 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:15.113341 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:15.113308 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs podName:d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb nodeName:}" failed. No retries permitted until 2026-07-30 10:48:16.113293919 +0000 UTC m=+4.086154930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs") pod "network-metrics-daemon-chjhd" (UID: "d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:15.213674 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.213636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8pc\" (UniqueName: \"kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc\") pod \"network-check-target-cdb5g\" (UID: \"d86e960e-cae8-4510-afcd-37dc31b105e6\") " pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:15.213846 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:15.213758 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 10:48:15.213846 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:15.213772 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 10:48:15.213846 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:15.213780 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hb8pc for pod openshift-network-diagnostics/network-check-target-cdb5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:15.213846 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:15.213836 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc podName:d86e960e-cae8-4510-afcd-37dc31b105e6 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:16.213823212 +0000 UTC m=+4.186684223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb8pc" (UniqueName: "kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc") pod "network-check-target-cdb5g" (UID: "d86e960e-cae8-4510-afcd-37dc31b105e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:15.424810 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:15.424776 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca678be3_1a7c_41dc_adb3_7fc44a1887ed.slice/crio-559c9a9dacc2eb50a2ea8233d5825f6724c3be471833093f975f0eeae7aef414 WatchSource:0}: Error finding container 559c9a9dacc2eb50a2ea8233d5825f6724c3be471833093f975f0eeae7aef414: Status 404 returned error can't find the container with id 559c9a9dacc2eb50a2ea8233d5825f6724c3be471833093f975f0eeae7aef414 Jul 30 10:48:15.427508 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:15.427148 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95bb37e_4aed_48e4_bbf6_b152ca7b685d.slice/crio-becc0dfd8f8f37744a249be35567f186ba3f7fec0b4ac3704937c3ad1a02e80e WatchSource:0}: Error finding container becc0dfd8f8f37744a249be35567f186ba3f7fec0b4ac3704937c3ad1a02e80e: Status 404 returned error can't find the container with id becc0dfd8f8f37744a249be35567f186ba3f7fec0b4ac3704937c3ad1a02e80e Jul 30 10:48:15.429659 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:15.429578 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177e8448_e3d4_40b5_aa14_9889d7901b64.slice/crio-cc02c42b6ac8ef488ae5081dc1e876f0b5ba82130ac7a21764044b90d87ab3b0 WatchSource:0}: Error finding container cc02c42b6ac8ef488ae5081dc1e876f0b5ba82130ac7a21764044b90d87ab3b0: Status 404 returned error can't find the container with id cc02c42b6ac8ef488ae5081dc1e876f0b5ba82130ac7a21764044b90d87ab3b0 Jul 30 10:48:15.430278 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:15.430251 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125f3a8d_6f3a_434c_b8d7_45d9f56b32e5.slice/crio-66aef9412717997f14c5ec35cfdea1bb0870967f8c07b8c9fc58a5c7e902c41e WatchSource:0}: Error finding container 66aef9412717997f14c5ec35cfdea1bb0870967f8c07b8c9fc58a5c7e902c41e: Status 404 returned error can't find the container with id 66aef9412717997f14c5ec35cfdea1bb0870967f8c07b8c9fc58a5c7e902c41e Jul 30 10:48:15.431604 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:15.431579 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c00f42_bdbc_49fe_9c83_e186f54e6ac8.slice/crio-0072f4a5b56e90d5cfab9a05a16527cf1b7fcb3e50454141b8e566bb5b35dc3e WatchSource:0}: Error finding container 0072f4a5b56e90d5cfab9a05a16527cf1b7fcb3e50454141b8e566bb5b35dc3e: Status 404 returned error can't find the container with id 0072f4a5b56e90d5cfab9a05a16527cf1b7fcb3e50454141b8e566bb5b35dc3e Jul 30 10:48:15.432528 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:15.432499 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd19302d_83a0_404d_923b_0ed52e8bcf51.slice/crio-e18f5a560b6c3240df115c7e4737518b95a287ef533aff8a183c70aba6e9c138 WatchSource:0}: Error finding container e18f5a560b6c3240df115c7e4737518b95a287ef533aff8a183c70aba6e9c138: Status 404 returned error can't find the container with id e18f5a560b6c3240df115c7e4737518b95a287ef533aff8a183c70aba6e9c138 Jul 30 10:48:15.433792 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:15.433311 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8307c4d_a9aa_4e53_9b75_ca7bc4080028.slice/crio-5881da1565b0423ac4d96f02ae84d7b037fa3cf555fa752d855810d1961ea229 WatchSource:0}: Error finding container 5881da1565b0423ac4d96f02ae84d7b037fa3cf555fa752d855810d1961ea229: Status 404 returned error can't find the container with id 5881da1565b0423ac4d96f02ae84d7b037fa3cf555fa752d855810d1961ea229 Jul 30 10:48:15.434435 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:15.434241 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7bd26f_41bc_4fda_8adc_f485de7344bb.slice/crio-a314f5046808d7ddf7d15a27dccf464d1e039a43ad29b8e3e16c6a6371fa51ae WatchSource:0}: Error finding container a314f5046808d7ddf7d15a27dccf464d1e039a43ad29b8e3e16c6a6371fa51ae: Status 404 returned error can't find the container with id a314f5046808d7ddf7d15a27dccf464d1e039a43ad29b8e3e16c6a6371fa51ae Jul 30 10:48:15.436041 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:15.436017 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc50509_0334_4ee6_b2c8_ae756587b675.slice/crio-7d8b1be599bbbd48faf02633e246aa6a9e35c64c3b9db7fa1a9688fcad5431f9 WatchSource:0}: Error finding container 7d8b1be599bbbd48faf02633e246aa6a9e35c64c3b9db7fa1a9688fcad5431f9: Status 404 returned error can't find the container with id 7d8b1be599bbbd48faf02633e246aa6a9e35c64c3b9db7fa1a9688fcad5431f9 Jul 30 10:48:15.449689 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.449645 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 10:43:14 +0000 UTC" deadline="2028-03-02 03:27:42.495183574 +0000 UTC" Jul 30 10:48:15.449689 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.449686 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13936h39m27.045502608s" Jul 30 10:48:15.530632 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.530446 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" event={"ID":"3e7bd26f-41bc-4fda-8adc-f485de7344bb","Type":"ContainerStarted","Data":"a314f5046808d7ddf7d15a27dccf464d1e039a43ad29b8e3e16c6a6371fa51ae"} Jul 30 10:48:15.531563 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.531541 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f5vrl" event={"ID":"f8307c4d-a9aa-4e53-9b75-ca7bc4080028","Type":"ContainerStarted","Data":"5881da1565b0423ac4d96f02ae84d7b037fa3cf555fa752d855810d1961ea229"} Jul 30 10:48:15.533459 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.533439 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dx77j" event={"ID":"dd19302d-83a0-404d-923b-0ed52e8bcf51","Type":"ContainerStarted","Data":"e18f5a560b6c3240df115c7e4737518b95a287ef533aff8a183c70aba6e9c138"} Jul 30 10:48:15.534682 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.534661 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b8skc" event={"ID":"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8","Type":"ContainerStarted","Data":"0072f4a5b56e90d5cfab9a05a16527cf1b7fcb3e50454141b8e566bb5b35dc3e"} Jul 30 10:48:15.535866 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.535836 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" event={"ID":"ffc50509-0334-4ee6-b2c8-ae756587b675","Type":"ContainerStarted","Data":"7d8b1be599bbbd48faf02633e246aa6a9e35c64c3b9db7fa1a9688fcad5431f9"} Jul 30 10:48:15.537077 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.537048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dspfd" event={"ID":"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5","Type":"ContainerStarted","Data":"66aef9412717997f14c5ec35cfdea1bb0870967f8c07b8c9fc58a5c7e902c41e"} Jul 30 10:48:15.538012 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.537986 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qwpkd" event={"ID":"177e8448-e3d4-40b5-aa14-9889d7901b64","Type":"ContainerStarted","Data":"cc02c42b6ac8ef488ae5081dc1e876f0b5ba82130ac7a21764044b90d87ab3b0"} Jul 30 10:48:15.539034 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.539011 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" event={"ID":"c95bb37e-4aed-48e4-bbf6-b152ca7b685d","Type":"ContainerStarted","Data":"becc0dfd8f8f37744a249be35567f186ba3f7fec0b4ac3704937c3ad1a02e80e"} Jul 30 10:48:15.540202 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:15.540181 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8kwj2" event={"ID":"ca678be3-1a7c-41dc-adb3-7fc44a1887ed","Type":"ContainerStarted","Data":"559c9a9dacc2eb50a2ea8233d5825f6724c3be471833093f975f0eeae7aef414"} Jul 30 10:48:16.019787 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.019745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:16.019977 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.019935 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:16.020046 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.020015 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret podName:f31dbccc-5060-4cb2-9df5-279124fcd806 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:18.019993108 +0000 UTC m=+5.992854136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret") pod "global-pull-secret-syncer-452jf" (UID: "f31dbccc-5060-4cb2-9df5-279124fcd806") : object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:16.120170 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.120131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:16.120357 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.120327 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:16.120437 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.120393 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs podName:d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb nodeName:}" failed. No retries permitted until 2026-07-30 10:48:18.120375253 +0000 UTC m=+6.093236278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs") pod "network-metrics-daemon-chjhd" (UID: "d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:16.197229 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.197148 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 10:48:16.221504 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.221466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8pc\" (UniqueName: \"kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc\") pod \"network-check-target-cdb5g\" (UID: \"d86e960e-cae8-4510-afcd-37dc31b105e6\") " pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:16.221681 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.221657 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 10:48:16.221681 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.221677 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 10:48:16.221795 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.221691 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hb8pc for pod openshift-network-diagnostics/network-check-target-cdb5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:16.221795 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.221748 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc podName:d86e960e-cae8-4510-afcd-37dc31b105e6 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:18.221729742 +0000 UTC m=+6.194590772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb8pc" (UniqueName: "kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc") pod "network-check-target-cdb5g" (UID: "d86e960e-cae8-4510-afcd-37dc31b105e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:16.450521 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.450408 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 10:43:14 +0000 UTC" deadline="2028-04-27 22:56:31.644995538 +0000 UTC" Jul 30 10:48:16.450521 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.450442 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15300h8m15.194557669s" Jul 30 10:48:16.525167 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.523999 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:16.525167 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.524143 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:16.525167 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.524582 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:16.525167 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.524687 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:16.525167 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.524742 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:16.525167 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:16.524800 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:16.548294 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.548253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" event={"ID":"e190c0b1e0328f4dc4c1440290f7e5c1","Type":"ContainerStarted","Data":"2d9d45f620adf34d375dc0bf5ee4dff39894d1e86f16f7bdbaf7cc33fc5299fe"} Jul 30 10:48:16.563013 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.562982 2575 generic.go:356] "Generic (PLEG): container finished" podID="675b44794ae9b75b8a3551e08f825170" containerID="4ef9ac05e2b9edf2489e054b979b04215209fc5302f149316475d3142d4ac998" exitCode=0 Jul 30 10:48:16.563208 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.563039 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" event={"ID":"675b44794ae9b75b8a3551e08f825170","Type":"ContainerDied","Data":"4ef9ac05e2b9edf2489e054b979b04215209fc5302f149316475d3142d4ac998"} Jul 30 10:48:16.620172 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:16.619335 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-131.ec2.internal" podStartSLOduration=3.61931356 podStartE2EDuration="3.61931356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:16.586350779 +0000 UTC m=+4.559211829" watchObservedRunningTime="2026-07-30 10:48:16.61931356 +0000 UTC m=+4.592174595" Jul 30 10:48:17.583412 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:17.583166 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" event={"ID":"675b44794ae9b75b8a3551e08f825170","Type":"ContainerStarted","Data":"2db7fee1d336691ae524213abdaae56ee53291641d7ded31b22dbe7b44d4d8dd"} Jul 30 10:48:17.650455 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:17.649944 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-131.ec2.internal" podStartSLOduration=4.649921075 podStartE2EDuration="4.649921075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:17.649334333 +0000 UTC m=+5.622195369" watchObservedRunningTime="2026-07-30 10:48:17.649921075 +0000 UTC m=+5.622782114" Jul 30 10:48:18.037978 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:18.037937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:18.038188 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.038158 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:18.038279 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.038232 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret podName:f31dbccc-5060-4cb2-9df5-279124fcd806 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:22.038211318 +0000 UTC m=+10.011072350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret") pod "global-pull-secret-syncer-452jf" (UID: "f31dbccc-5060-4cb2-9df5-279124fcd806") : object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:18.139211 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:18.139169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:18.139408 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.139392 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:18.139470 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.139456 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs podName:d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb nodeName:}" failed. No retries permitted until 2026-07-30 10:48:22.139436003 +0000 UTC m=+10.112297023 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs") pod "network-metrics-daemon-chjhd" (UID: "d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:18.240234 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:18.240192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8pc\" (UniqueName: \"kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc\") pod \"network-check-target-cdb5g\" (UID: \"d86e960e-cae8-4510-afcd-37dc31b105e6\") " pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:18.240421 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.240369 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 10:48:18.240421 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.240390 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 10:48:18.240421 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.240403 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hb8pc for pod openshift-network-diagnostics/network-check-target-cdb5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:18.240593 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.240465 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc podName:d86e960e-cae8-4510-afcd-37dc31b105e6 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:22.240445589 +0000 UTC m=+10.213306604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb8pc" (UniqueName: "kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc") pod "network-check-target-cdb5g" (UID: "d86e960e-cae8-4510-afcd-37dc31b105e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:18.523452 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:18.523209 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:18.523452 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:18.523221 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:18.523452 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.523365 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:18.523452 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.523434 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:18.523763 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:18.523495 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:18.523763 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:18.523568 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:20.521297 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:20.521264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:20.521731 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:20.521303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:20.521731 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:20.521264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:20.521731 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:20.521403 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:20.521731 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:20.521509 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:20.521731 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:20.521612 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:22.071396 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:22.071356 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:22.071846 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.071550 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:22.071846 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.071622 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret podName:f31dbccc-5060-4cb2-9df5-279124fcd806 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:30.071602956 +0000 UTC m=+18.044463970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret") pod "global-pull-secret-syncer-452jf" (UID: "f31dbccc-5060-4cb2-9df5-279124fcd806") : object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:22.171751 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:22.171717 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:22.171947 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.171834 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:22.171947 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.171883 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs podName:d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb nodeName:}" failed. No retries permitted until 2026-07-30 10:48:30.171869163 +0000 UTC m=+18.144730175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs") pod "network-metrics-daemon-chjhd" (UID: "d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:22.272734 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:22.272689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8pc\" (UniqueName: \"kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc\") pod \"network-check-target-cdb5g\" (UID: \"d86e960e-cae8-4510-afcd-37dc31b105e6\") " pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:22.272917 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.272868 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 10:48:22.272917 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.272895 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 10:48:22.272917 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.272908 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hb8pc for pod openshift-network-diagnostics/network-check-target-cdb5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:22.273070 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.272976 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc podName:d86e960e-cae8-4510-afcd-37dc31b105e6 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:30.272954969 +0000 UTC m=+18.245815990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb8pc" (UniqueName: "kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc") pod "network-check-target-cdb5g" (UID: "d86e960e-cae8-4510-afcd-37dc31b105e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:22.522592 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:22.522503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:22.522747 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.522632 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:22.522905 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:22.522884 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:22.522970 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:22.522946 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:22.523049 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.523030 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:22.523199 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:22.523164 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:24.521631 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:24.521597 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:24.522130 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:24.521740 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:24.522205 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:24.522146 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:24.522205 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:24.522173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:24.522319 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:24.522250 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:24.522373 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:24.522333 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:26.521119 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:26.521018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:26.521568 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:26.521171 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:26.521568 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:26.521018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:26.521568 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:26.521271 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:26.521568 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:26.521018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:26.521568 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:26.521320 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:28.521001 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:28.520963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:28.521517 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:28.521019 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:28.521517 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:28.520963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:28.521517 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:28.521136 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:28.521517 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:28.521259 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:28.521517 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:28.521365 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:30.126137 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:30.126068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:30.126655 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.126232 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:30.126655 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.126323 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret podName:f31dbccc-5060-4cb2-9df5-279124fcd806 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:46.126298592 +0000 UTC m=+34.099159623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret") pod "global-pull-secret-syncer-452jf" (UID: "f31dbccc-5060-4cb2-9df5-279124fcd806") : object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:30.226664 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:30.226620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:30.226854 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.226784 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:30.226910 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.226860 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs podName:d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb nodeName:}" failed. No retries permitted until 2026-07-30 10:48:46.226838509 +0000 UTC m=+34.199699543 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs") pod "network-metrics-daemon-chjhd" (UID: "d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:30.327793 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:30.327757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8pc\" (UniqueName: \"kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc\") pod \"network-check-target-cdb5g\" (UID: \"d86e960e-cae8-4510-afcd-37dc31b105e6\") " pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:30.327980 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.327919 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 10:48:30.327980 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.327939 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 10:48:30.327980 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.327952 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hb8pc for pod openshift-network-diagnostics/network-check-target-cdb5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:30.328144 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.328017 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc podName:d86e960e-cae8-4510-afcd-37dc31b105e6 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:46.32799842 +0000 UTC m=+34.300859452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb8pc" (UniqueName: "kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc") pod "network-check-target-cdb5g" (UID: "d86e960e-cae8-4510-afcd-37dc31b105e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:30.521431 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:30.521344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:30.521603 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:30.521344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:30.521603 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.521494 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:30.521603 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:30.521353 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:30.521603 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.521559 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:30.521808 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:30.521611 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:32.521588 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:32.521553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:32.522074 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:32.521673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:32.522074 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:32.521679 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:32.522074 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:32.521707 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:32.522074 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:32.521815 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:32.522074 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:32.521894 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:34.521288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.521106 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:34.522080 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.521135 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:34.522080 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.521140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:34.522080 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:34.521399 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:34.522080 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:34.521516 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:34.522080 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:34.521639 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:34.618710 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.618675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" event={"ID":"ffc50509-0334-4ee6-b2c8-ae756587b675","Type":"ContainerStarted","Data":"8ad8a793aecb478cbaa8bc67bd6df0eb9fe63161d9b4caaee900cad6f54f82e0"} Jul 30 10:48:34.618896 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.618749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" event={"ID":"ffc50509-0334-4ee6-b2c8-ae756587b675","Type":"ContainerStarted","Data":"148cd0b912fca36c475c576ebd6da8d2cfcd7139ad58879702748f31619e75c1"} Jul 30 10:48:34.618896 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.618782 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" event={"ID":"ffc50509-0334-4ee6-b2c8-ae756587b675","Type":"ContainerStarted","Data":"7be21f5710d6aac195dd7f87152908fd7b521191663710a45b171b8d87617cbf"} Jul 30 10:48:34.620185 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.620061 2575 generic.go:356] "Generic (PLEG): container finished" podID="125f3a8d-6f3a-434c-b8d7-45d9f56b32e5" containerID="0d23b0d7d0d8ab12fc705d3a2b23629324e932c910f04f82eba6052c616ef495" exitCode=0 Jul 30 10:48:34.620185 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.620176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dspfd" event={"ID":"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5","Type":"ContainerDied","Data":"0d23b0d7d0d8ab12fc705d3a2b23629324e932c910f04f82eba6052c616ef495"} Jul 30 10:48:34.621683 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.621629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qwpkd" event={"ID":"177e8448-e3d4-40b5-aa14-9889d7901b64","Type":"ContainerStarted","Data":"81b83fb1a92017dafb141d02c8e6dc6055ec75d539e0eed3c88229e789e52277"} Jul 30 10:48:34.623190 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.623152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" event={"ID":"c95bb37e-4aed-48e4-bbf6-b152ca7b685d","Type":"ContainerStarted","Data":"0c0f868c86830d57b899f08b42aa435afdcca53f53c4bbc35f61db7cbd83bc5d"} Jul 30 10:48:34.624565 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.624537 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8kwj2" event={"ID":"ca678be3-1a7c-41dc-adb3-7fc44a1887ed","Type":"ContainerStarted","Data":"3160c81f06ee129605c639f9d7a4b72b695770fe95623278bd47f37062398125"} Jul 30 10:48:34.626296 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.626208 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" event={"ID":"3e7bd26f-41bc-4fda-8adc-f485de7344bb","Type":"ContainerStarted","Data":"84dde24ce83385597fd99c5cd9945e556e53ff247d596025ff5f3a52bb3b0082"} Jul 30 10:48:34.627699 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.627675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f5vrl" event={"ID":"f8307c4d-a9aa-4e53-9b75-ca7bc4080028","Type":"ContainerStarted","Data":"41626c8b17ca19dfb5312f8553b8ea67df0d8277927d5c8541615cef15ff54d2"} Jul 30 10:48:34.628973 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.628951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dx77j" event={"ID":"dd19302d-83a0-404d-923b-0ed52e8bcf51","Type":"ContainerStarted","Data":"8dfef7af4b5c8ff803b01bd569886d635eb0f9e5a7a27714c2dad257a3f53c1b"} Jul 30 10:48:34.659141 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.659074 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qwpkd" podStartSLOduration=3.938553257 podStartE2EDuration="22.659044495s" totalImagesPullingTime="18.720491238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:34.658860052 +0000 UTC m=+22.631721088" watchObservedRunningTime="2026-07-30 10:48:34.659044495 +0000 UTC m=+22.631905531" Jul 30 10:48:34.696251 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.696196 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f5vrl" podStartSLOduration=3.979722913 podStartE2EDuration="22.696179364s" totalImagesPullingTime="18.716456451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:34.696149541 +0000 UTC m=+22.669010588" watchObservedRunningTime="2026-07-30 10:48:34.696179364 +0000 UTC m=+22.669040399" Jul 30 10:48:34.696390 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.696311 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8kwj2" podStartSLOduration=3.93520411 podStartE2EDuration="22.696305704s" totalImagesPullingTime="18.761101594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:34.677496622 +0000 UTC m=+22.650357658" watchObservedRunningTime="2026-07-30 10:48:34.696305704 +0000 UTC m=+22.669166739" Jul 30 10:48:34.717553 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.717503 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dx77j" podStartSLOduration=4.285965051 podStartE2EDuration="22.717484484s" totalImagesPullingTime="18.431519433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:34.717380315 +0000 UTC m=+22.690241350" watchObservedRunningTime="2026-07-30 10:48:34.717484484 +0000 UTC m=+22.690345519" Jul 30 10:48:34.736192 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:34.736133 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nzkqn" podStartSLOduration=4.029228338 podStartE2EDuration="22.736110261s" totalImagesPullingTime="18.706881923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:34.734994689 +0000 UTC m=+22.707855727" watchObservedRunningTime="2026-07-30 10:48:34.736110261 +0000 UTC m=+22.708971288" Jul 30 10:48:35.632862 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:35.632801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b8skc" event={"ID":"f8c00f42-bdbc-49fe-9c83-e186f54e6ac8","Type":"ContainerStarted","Data":"1f1f6fea250d9569ec91c5e1db97882f3e71092b05fae5e77ff0781e9041693b"} Jul 30 10:48:35.636431 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:35.636401 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" event={"ID":"ffc50509-0334-4ee6-b2c8-ae756587b675","Type":"ContainerStarted","Data":"b138deefeb1fe5fe73acf3c45d6c45c998d226cb0fd17dc0a038dc7f26e72ba2"} Jul 30 10:48:35.636607 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:35.636487 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" event={"ID":"ffc50509-0334-4ee6-b2c8-ae756587b675","Type":"ContainerStarted","Data":"e76a3a245380bb6b77c423ff10236f8f6c9d19481aca0d7de65a05c3df9d37b9"} Jul 30 10:48:35.636607 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:35.636536 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" event={"ID":"ffc50509-0334-4ee6-b2c8-ae756587b675","Type":"ContainerStarted","Data":"612a6133ef260902e7c478f880ee6e275c08173fef39e01603e18bbc240ada64"} Jul 30 10:48:35.648479 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:35.648429 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b8skc" podStartSLOduration=4.909199333 podStartE2EDuration="23.648412186s" totalImagesPullingTime="18.739212853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:35.648118281 +0000 UTC m=+23.620979334" watchObservedRunningTime="2026-07-30 10:48:35.648412186 +0000 UTC m=+23.621273221" Jul 30 10:48:35.796960 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:35.796884 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 10:48:36.158344 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:36.158262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:36.463261 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:36.463105 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T10:48:35.796909328Z","UUID":"684edb2c-1de7-492c-bdc6-351f6e4ed13d","Handler":null,"Name":"","Endpoint":""} Jul 30 10:48:36.464901 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:36.464873 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 10:48:36.465049 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:36.464920 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 10:48:36.521015 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:36.520984 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:36.521226 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:36.520984 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:36.521226 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:36.521141 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:36.521226 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:36.520987 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:36.521393 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:36.521225 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:36.521393 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:36.521333 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:36.640903 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:36.640870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" event={"ID":"c95bb37e-4aed-48e4-bbf6-b152ca7b685d","Type":"ContainerStarted","Data":"962dc1866499719347817db9b5950024c95e8414c413cf316aaf16b59ffce907"} Jul 30 10:48:37.650523 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:37.650482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" event={"ID":"ffc50509-0334-4ee6-b2c8-ae756587b675","Type":"ContainerStarted","Data":"e063e76603dc37f5582d7f21d607c7a079b6f556fd9b828eca7c8d99e6a03fcb"} Jul 30 10:48:37.854609 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:37.854561 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:37.855183 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:37.855161 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:38.521776 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:38.521605 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:38.521943 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:38.521605 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:38.521943 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:38.521853 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:38.521943 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:38.521606 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:38.521943 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:38.521926 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:38.522117 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:38.522001 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:38.653376 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:38.653340 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dx77j" Jul 30 10:48:39.658077 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:39.658001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" event={"ID":"ffc50509-0334-4ee6-b2c8-ae756587b675","Type":"ContainerStarted","Data":"fa0ee11864ae22fefd221ac8a03b4faefde20ae50f2691831625130a5e38c661"} Jul 30 10:48:39.658928 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:39.658322 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:39.660007 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:39.659981 2575 generic.go:356] "Generic (PLEG): container finished" podID="125f3a8d-6f3a-434c-b8d7-45d9f56b32e5" containerID="400206e7d21e1d536fd1d6f18038dbc27e4e534b565321abacbef2e8a6206934" exitCode=0 Jul 30 10:48:39.660150 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:39.660058 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dspfd" event={"ID":"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5","Type":"ContainerDied","Data":"400206e7d21e1d536fd1d6f18038dbc27e4e534b565321abacbef2e8a6206934"} Jul 30 10:48:39.674077 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:39.674055 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:39.684046 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:39.684005 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" podStartSLOduration=8.895845401999999 podStartE2EDuration="27.683990385s" totalImagesPullingTime="18.788144983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:39.683612167 +0000 UTC m=+27.656473203" watchObservedRunningTime="2026-07-30 10:48:39.683990385 +0000 UTC m=+27.656851419" Jul 30 10:48:40.521431 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:40.521193 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:40.521635 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:40.521434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:40.521635 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:40.521546 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:40.521975 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:40.521946 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:40.522035 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:40.522013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:40.522110 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:40.522080 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:40.664255 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:40.664176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" event={"ID":"c95bb37e-4aed-48e4-bbf6-b152ca7b685d","Type":"ContainerStarted","Data":"b5e4ded25dac11c1aeb5bd83d2c75cfcb3a39b3547886d9b2ae55baf8d35e041"} Jul 30 10:48:40.664645 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:40.664414 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:40.664645 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:40.664484 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:40.679501 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:40.679474 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:48:40.687512 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:40.687470 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m8jdm" podStartSLOduration=4.714171289 podStartE2EDuration="28.68745667s" totalImagesPullingTime="23.973285381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:40.686769203 +0000 UTC m=+28.659630248" watchObservedRunningTime="2026-07-30 10:48:40.68745667 +0000 UTC m=+28.660317753" Jul 30 10:48:41.462227 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:41.462195 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-chjhd"] Jul 30 10:48:41.462448 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:41.462320 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:41.462448 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:41.462417 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:41.469280 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:41.469228 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-452jf"] Jul 30 10:48:41.469473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:41.469316 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:41.469473 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:41.469394 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:41.469773 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:41.469754 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cdb5g"] Jul 30 10:48:41.469835 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:41.469825 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:41.469925 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:41.469894 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:41.668570 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:41.668538 2575 generic.go:356] "Generic (PLEG): container finished" podID="125f3a8d-6f3a-434c-b8d7-45d9f56b32e5" containerID="e38144119a206f0f0b635d51b9ae2cf689e5ac96d0c39c98087862de0d3e27d7" exitCode=0 Jul 30 10:48:41.669052 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:41.668620 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dspfd" event={"ID":"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5","Type":"ContainerDied","Data":"e38144119a206f0f0b635d51b9ae2cf689e5ac96d0c39c98087862de0d3e27d7"} Jul 30 10:48:43.521031 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:43.520867 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:43.521403 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:43.520867 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:43.521403 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:43.521130 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:43.521403 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:43.520912 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:43.521403 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:43.521220 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:43.521403 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:43.521279 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:43.674995 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:43.674960 2575 generic.go:356] "Generic (PLEG): container finished" podID="125f3a8d-6f3a-434c-b8d7-45d9f56b32e5" containerID="cc873f41934c4af05905466e3bbbd8df6e7aeb0d197f746137a6539f7aa8568f" exitCode=0 Jul 30 10:48:43.675218 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:43.675039 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dspfd" event={"ID":"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5","Type":"ContainerDied","Data":"cc873f41934c4af05905466e3bbbd8df6e7aeb0d197f746137a6539f7aa8568f"} Jul 30 10:48:45.521135 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:45.521025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:45.521135 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:45.521064 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:45.521135 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:45.521064 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:45.521820 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:45.521160 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cdb5g" podUID="d86e960e-cae8-4510-afcd-37dc31b105e6" Jul 30 10:48:45.521820 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:45.521269 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-452jf" podUID="f31dbccc-5060-4cb2-9df5-279124fcd806" Jul 30 10:48:45.521820 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:45.521336 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-chjhd" podUID="d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb" Jul 30 10:48:46.144717 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.144682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:46.144890 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:46.144809 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:46.144890 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:46.144869 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret podName:f31dbccc-5060-4cb2-9df5-279124fcd806 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:18.144850121 +0000 UTC m=+66.117711136 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret") pod "global-pull-secret-syncer-452jf" (UID: "f31dbccc-5060-4cb2-9df5-279124fcd806") : object "kube-system"/"original-pull-secret" not registered Jul 30 10:48:46.245202 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.245169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:46.245376 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:46.245328 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:46.245422 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:46.245407 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs podName:d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb nodeName:}" failed. No retries permitted until 2026-07-30 10:49:18.245388734 +0000 UTC m=+66.218249747 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs") pod "network-metrics-daemon-chjhd" (UID: "d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 10:48:46.346430 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.346390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8pc\" (UniqueName: \"kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc\") pod \"network-check-target-cdb5g\" (UID: \"d86e960e-cae8-4510-afcd-37dc31b105e6\") " pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:46.346597 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:46.346559 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 10:48:46.346597 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:46.346581 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 10:48:46.346597 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:46.346591 2575 projected.go:196] Error preparing data for projected volume kube-api-access-hb8pc for pod openshift-network-diagnostics/network-check-target-cdb5g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:46.346794 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:46.346651 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc podName:d86e960e-cae8-4510-afcd-37dc31b105e6 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:18.34663156 +0000 UTC m=+66.319492581 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb8pc" (UniqueName: "kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc") pod "network-check-target-cdb5g" (UID: "d86e960e-cae8-4510-afcd-37dc31b105e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 10:48:46.849728 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.849697 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-131.ec2.internal" event="NodeReady" Jul 30 10:48:46.850182 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.849831 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 10:48:46.900036 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.900006 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mpcdj"] Jul 30 10:48:46.919244 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.919019 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj"] Jul 30 10:48:46.919244 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.919174 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:46.922264 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.922160 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 30 10:48:46.922780 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.922757 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 30 10:48:46.923136 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.923110 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 30 10:48:46.923350 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.923336 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g9dc8\"" Jul 30 10:48:46.923484 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.923464 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 30 10:48:46.931227 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.930993 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 30 10:48:46.937046 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.936969 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-h8zf7"] Jul 30 10:48:46.937189 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.937114 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:48:46.939386 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.939294 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 30 10:48:46.939386 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.939373 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 30 10:48:46.939571 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.939412 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g9vp6\"" Jul 30 10:48:46.939571 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.939443 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 30 10:48:46.961375 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.961344 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bcb89d8c9-shq24"] Jul 30 10:48:46.961552 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.961470 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:46.963403 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.963353 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 30 10:48:46.963597 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.963550 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 30 10:48:46.963721 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.963610 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-s4pdq\"" Jul 30 10:48:46.964023 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.964006 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 10:48:46.964396 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.964377 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 10:48:46.973850 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.973828 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 30 10:48:46.975607 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.975582 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r"] Jul 30 10:48:46.975741 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.975722 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:46.977933 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.977909 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 30 10:48:46.978043 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.977962 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 30 10:48:46.978043 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.977934 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpkjw\"" Jul 30 10:48:46.978343 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.978321 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 30 10:48:46.984096 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.984069 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 30 10:48:46.987452 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.987428 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mpcdj"] Jul 30 10:48:46.987569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.987460 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-h8zf7"] Jul 30 10:48:46.987569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.987479 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj"] Jul 30 10:48:46.987569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.987505 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r"] Jul 30 10:48:46.987569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.987523 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bcb89d8c9-shq24"] Jul 30 10:48:46.987569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.987543 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-msgck"] Jul 30 10:48:46.987789 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.987580 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:46.990665 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.990637 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 10:48:46.990775 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.990644 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 10:48:46.991147 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.991127 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 30 10:48:46.991255 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.991153 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vdjzf\"" Jul 30 10:48:46.991334 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:46.991311 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 30 10:48:47.001672 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.001644 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms"] Jul 30 10:48:47.001811 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.001747 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.004263 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.004237 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 10:48:47.004425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.004237 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6l9vk\"" Jul 30 10:48:47.004681 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.004663 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 10:48:47.015721 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.015697 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8"] Jul 30 10:48:47.015852 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.015819 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.025870 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.025848 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5984b4bcb7-nxpbf"] Jul 30 10:48:47.025996 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.025978 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8" Jul 30 10:48:47.027610 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.027462 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 30 10:48:47.027610 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.027473 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 30 10:48:47.027610 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.027537 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 30 10:48:47.027816 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.027791 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mnf8h\"" Jul 30 10:48:47.027926 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.027909 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 30 10:48:47.030079 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.030046 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 30 10:48:47.030211 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.030159 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-n9jqg\"" Jul 30 10:48:47.030504 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.030484 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 30 10:48:47.038083 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.038055 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2"] Jul 30 10:48:47.038227 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.038211 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.041597 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.041571 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 30 10:48:47.041597 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.041596 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 30 10:48:47.041791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.041618 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 30 10:48:47.041791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.041672 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 30 10:48:47.041791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.041720 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 30 10:48:47.041791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.041631 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-45qtj\"" Jul 30 10:48:47.042011 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.041994 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 30 10:48:47.052219 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-installation-pull-secrets\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.052219 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b8d2abb-c386-4bca-9278-c37aff6403fa-config\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.052420 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-image-registry-private-configuration\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.052420 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/829982dc-21f7-4ee7-accb-0182ccd29add-serving-cert\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.052420 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052325 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpsjz\" (UniqueName: \"kubernetes.io/projected/829982dc-21f7-4ee7-accb-0182ccd29add-kube-api-access-fpsjz\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.052420 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkdg\" (UniqueName: \"kubernetes.io/projected/8c4dfaea-cd20-41df-90ea-0a961a08363d-kube-api-access-mxkdg\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:48:47.052609 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98410b9f-a736-487f-8c82-b3df27f5113b-ca-trust-extracted\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.052609 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829982dc-21f7-4ee7-accb-0182ccd29add-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.052609 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-tmp-dir\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.052609 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052512 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpdsr\" (UniqueName: \"kubernetes.io/projected/9b8d2abb-c386-4bca-9278-c37aff6403fa-kube-api-access-fpdsr\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.052609 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czhjf\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-kube-api-access-czhjf\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.052609 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b8d2abb-c386-4bca-9278-c37aff6403fa-serving-cert\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.052823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052609 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:47.052823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29gk\" (UniqueName: \"kubernetes.io/projected/9a6ca007-2d7e-45d8-861e-79c9b7d07655-kube-api-access-n29gk\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:47.052823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-config-volume\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.052823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/829982dc-21f7-4ee7-accb-0182ccd29add-tmp\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.052823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.052823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-trusted-ca\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.052823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.053052 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052857 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829982dc-21f7-4ee7-accb-0182ccd29add-service-ca-bundle\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.053052 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2krs8\" (UniqueName: \"kubernetes.io/projected/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-kube-api-access-2krs8\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.053052 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2abb-c386-4bca-9278-c37aff6403fa-trusted-ca\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.053052 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.052993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9a6ca007-2d7e-45d8-861e-79c9b7d07655-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:47.053052 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.053011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-registry-certificates\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.053052 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.053038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/829982dc-21f7-4ee7-accb-0182ccd29add-snapshots\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.053302 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.053121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:48:47.053302 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.053223 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-bound-sa-token\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.055589 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.055567 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx"] Jul 30 10:48:47.055732 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.055709 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2" Jul 30 10:48:47.058271 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.058249 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 10:48:47.058271 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.058249 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mcq64\"" Jul 30 10:48:47.058466 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.058249 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 10:48:47.067966 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.067941 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b5vtk"] Jul 30 10:48:47.068122 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.068081 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.071985 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.071962 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 30 10:48:47.072459 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.072436 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bzq8t\"" Jul 30 10:48:47.072553 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.072518 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 30 10:48:47.073016 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.073000 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 30 10:48:47.073762 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.073741 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 30 10:48:47.083674 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.083641 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-msgck"] Jul 30 10:48:47.083802 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.083687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms"] Jul 30 10:48:47.083802 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.083708 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8"] Jul 30 10:48:47.083802 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.083723 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2"] Jul 30 10:48:47.083802 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.083740 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx"] Jul 30 10:48:47.083802 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.083749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b5vtk" Jul 30 10:48:47.084057 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.083755 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b5vtk"] Jul 30 10:48:47.084057 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.083862 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5984b4bcb7-nxpbf"] Jul 30 10:48:47.088534 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.088489 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 10:48:47.088534 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.088492 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 10:48:47.088727 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.088631 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 10:48:47.088814 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.088797 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6r58j\"" Jul 30 10:48:47.154138 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-config-volume\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.154369 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/829982dc-21f7-4ee7-accb-0182ccd29add-tmp\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.154517 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d5c371a-680e-4c9b-aac7-aa5c13018096-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx\" (UID: \"7d5c371a-680e-4c9b-aac7-aa5c13018096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.154641 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-stats-auth\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.154641 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.154641 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-trusted-ca\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.154782 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.154782 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829982dc-21f7-4ee7-accb-0182ccd29add-service-ca-bundle\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.154782 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154709 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-config-volume\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.154782 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2krs8\" (UniqueName: \"kubernetes.io/projected/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-kube-api-access-2krs8\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.154782 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7xw\" (UniqueName: \"kubernetes.io/projected/5bc5e620-0946-4c46-8fa7-c27d2ee33343-kube-api-access-6d7xw\") pod \"ingress-canary-b5vtk\" (UID: \"5bc5e620-0946-4c46-8fa7-c27d2ee33343\") " pod="openshift-ingress-canary/ingress-canary-b5vtk" Jul 30 10:48:47.155065 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.154801 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 10:48:47.155065 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2abb-c386-4bca-9278-c37aff6403fa-trusted-ca\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.155065 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.154817 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcb89d8c9-shq24: secret "image-registry-tls" not found Jul 30 10:48:47.155065 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9a6ca007-2d7e-45d8-861e-79c9b7d07655-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:47.155065 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/829982dc-21f7-4ee7-accb-0182ccd29add-tmp\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.155065 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.154857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-registry-certificates\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.155065 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.154869 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls podName:98410b9f-a736-487f-8c82-b3df27f5113b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:47.654852786 +0000 UTC m=+35.627713813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls") pod "image-registry-7bcb89d8c9-shq24" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b") : secret "image-registry-tls" not found Jul 30 10:48:47.155065 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.154801 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 10:48:47.155065 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.155051 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls podName:fc7c9219-d5ac-4629-9b4f-0dcba70f47cd nodeName:}" failed. No retries permitted until 2026-07-30 10:48:47.655034611 +0000 UTC m=+35.627895638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls") pod "dns-default-msgck" (UID: "fc7c9219-d5ac-4629-9b4f-0dcba70f47cd") : secret "dns-default-metrics-tls" not found Jul 30 10:48:47.155065 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/829982dc-21f7-4ee7-accb-0182ccd29add-snapshots\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d78e2d15-951a-4df2-afce-ce74eb971837-config\") pod \"service-ca-operator-5d6c75f84f-f98ms\" (UID: \"d78e2d15-951a-4df2-afce-ce74eb971837\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-bound-sa-token\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-installation-pull-secrets\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155246 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdb88\" (UniqueName: \"kubernetes.io/projected/211111e8-5f51-40fe-aaad-516c49d73212-kube-api-access-wdb88\") pod \"network-check-source-7d88cb55d8-5twc2\" (UID: \"211111e8-5f51-40fe-aaad-516c49d73212\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155273 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b8d2abb-c386-4bca-9278-c37aff6403fa-config\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-image-registry-private-configuration\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155332 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/829982dc-21f7-4ee7-accb-0182ccd29add-serving-cert\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155382 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829982dc-21f7-4ee7-accb-0182ccd29add-service-ca-bundle\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpsjz\" (UniqueName: \"kubernetes.io/projected/829982dc-21f7-4ee7-accb-0182ccd29add-kube-api-access-fpsjz\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155424 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87f2m\" (UniqueName: \"kubernetes.io/projected/d78e2d15-951a-4df2-afce-ce74eb971837-kube-api-access-87f2m\") pod \"service-ca-operator-5d6c75f84f-f98ms\" (UID: \"d78e2d15-951a-4df2-afce-ce74eb971837\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155428 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-registry-certificates\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.155482 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 10:48:47.155605 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.155530 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls podName:8c4dfaea-cd20-41df-90ea-0a961a08363d nodeName:}" failed. No retries permitted until 2026-07-30 10:48:47.655514325 +0000 UTC m=+35.628375350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls") pod "cluster-samples-operator-6748489656-m4mfj" (UID: "8c4dfaea-cd20-41df-90ea-0a961a08363d") : secret "samples-operator-tls" not found Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9a6ca007-2d7e-45d8-861e-79c9b7d07655-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/829982dc-21f7-4ee7-accb-0182ccd29add-snapshots\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155739 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b8d2abb-c386-4bca-9278-c37aff6403fa-trusted-ca\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-default-certificate\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155761 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-trusted-ca\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155790 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkdg\" (UniqueName: \"kubernetes.io/projected/8c4dfaea-cd20-41df-90ea-0a961a08363d-kube-api-access-mxkdg\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98410b9f-a736-487f-8c82-b3df27f5113b-ca-trust-extracted\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829982dc-21f7-4ee7-accb-0182ccd29add-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hm48\" (UniqueName: \"kubernetes.io/projected/7d5c371a-680e-4c9b-aac7-aa5c13018096-kube-api-access-8hm48\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx\" (UID: \"7d5c371a-680e-4c9b-aac7-aa5c13018096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155898 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bc5e620-0946-4c46-8fa7-c27d2ee33343-cert\") pod \"ingress-canary-b5vtk\" (UID: \"5bc5e620-0946-4c46-8fa7-c27d2ee33343\") " pod="openshift-ingress-canary/ingress-canary-b5vtk" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-tmp-dir\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155944 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5c371a-680e-4c9b-aac7-aa5c13018096-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx\" (UID: \"7d5c371a-680e-4c9b-aac7-aa5c13018096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpdsr\" (UniqueName: \"kubernetes.io/projected/9b8d2abb-c386-4bca-9278-c37aff6403fa-kube-api-access-fpdsr\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.155995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czhjf\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-kube-api-access-czhjf\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156169 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98410b9f-a736-487f-8c82-b3df27f5113b-ca-trust-extracted\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.156458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156248 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b8d2abb-c386-4bca-9278-c37aff6403fa-config\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156257 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-tmp-dir\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156309 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b8d2abb-c386-4bca-9278-c37aff6403fa-serving-cert\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d78e2d15-951a-4df2-afce-ce74eb971837-serving-cert\") pod \"service-ca-operator-5d6c75f84f-f98ms\" (UID: \"d78e2d15-951a-4df2-afce-ce74eb971837\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29gk\" (UniqueName: \"kubernetes.io/projected/9a6ca007-2d7e-45d8-861e-79c9b7d07655-kube-api-access-n29gk\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wtcj\" (UniqueName: \"kubernetes.io/projected/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-kube-api-access-6wtcj\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.156437 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzvtq\" (UniqueName: \"kubernetes.io/projected/5acafbf4-35f9-49f6-abc9-5238956900fb-kube-api-access-wzvtq\") pod \"volume-data-source-validator-549c6d54cf-8kxd8\" (UID: \"5acafbf4-35f9-49f6-abc9-5238956900fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8" Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.156484 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls podName:9a6ca007-2d7e-45d8-861e-79c9b7d07655 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:47.656470388 +0000 UTC m=+35.629331400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-fz54r" (UID: "9a6ca007-2d7e-45d8-861e-79c9b7d07655") : secret "cluster-monitoring-operator-tls" not found Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.157286 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.156677 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/829982dc-21f7-4ee7-accb-0182ccd29add-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.160981 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.160954 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b8d2abb-c386-4bca-9278-c37aff6403fa-serving-cert\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.161139 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.160992 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/829982dc-21f7-4ee7-accb-0182ccd29add-serving-cert\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.161139 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.160960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-installation-pull-secrets\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.161139 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.161021 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-image-registry-private-configuration\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.173882 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.173851 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpdsr\" (UniqueName: \"kubernetes.io/projected/9b8d2abb-c386-4bca-9278-c37aff6403fa-kube-api-access-fpdsr\") pod \"console-operator-584dcb6b54-mpcdj\" (UID: \"9b8d2abb-c386-4bca-9278-c37aff6403fa\") " pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.174112 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.174017 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czhjf\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-kube-api-access-czhjf\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.174922 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.174899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29gk\" (UniqueName: \"kubernetes.io/projected/9a6ca007-2d7e-45d8-861e-79c9b7d07655-kube-api-access-n29gk\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:47.175034 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.175019 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkdg\" (UniqueName: \"kubernetes.io/projected/8c4dfaea-cd20-41df-90ea-0a961a08363d-kube-api-access-mxkdg\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:48:47.175404 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.175383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpsjz\" (UniqueName: \"kubernetes.io/projected/829982dc-21f7-4ee7-accb-0182ccd29add-kube-api-access-fpsjz\") pod \"insights-operator-5885fdb759-h8zf7\" (UID: \"829982dc-21f7-4ee7-accb-0182ccd29add\") " pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.175618 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.175593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-bound-sa-token\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.186849 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.186818 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2krs8\" (UniqueName: \"kubernetes.io/projected/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-kube-api-access-2krs8\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.246475 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.246395 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:47.257884 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.257846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d78e2d15-951a-4df2-afce-ce74eb971837-serving-cert\") pod \"service-ca-operator-5d6c75f84f-f98ms\" (UID: \"d78e2d15-951a-4df2-afce-ce74eb971837\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.258058 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.257894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wtcj\" (UniqueName: \"kubernetes.io/projected/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-kube-api-access-6wtcj\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.258058 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.257923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzvtq\" (UniqueName: \"kubernetes.io/projected/5acafbf4-35f9-49f6-abc9-5238956900fb-kube-api-access-wzvtq\") pod \"volume-data-source-validator-549c6d54cf-8kxd8\" (UID: \"5acafbf4-35f9-49f6-abc9-5238956900fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8" Jul 30 10:48:47.258058 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.257948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.258058 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.257987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d5c371a-680e-4c9b-aac7-aa5c13018096-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx\" (UID: \"7d5c371a-680e-4c9b-aac7-aa5c13018096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.258058 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-stats-auth\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.258331 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7xw\" (UniqueName: \"kubernetes.io/projected/5bc5e620-0946-4c46-8fa7-c27d2ee33343-kube-api-access-6d7xw\") pod \"ingress-canary-b5vtk\" (UID: \"5bc5e620-0946-4c46-8fa7-c27d2ee33343\") " pod="openshift-ingress-canary/ingress-canary-b5vtk" Jul 30 10:48:47.258331 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d78e2d15-951a-4df2-afce-ce74eb971837-config\") pod \"service-ca-operator-5d6c75f84f-f98ms\" (UID: \"d78e2d15-951a-4df2-afce-ce74eb971837\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.258331 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdb88\" (UniqueName: \"kubernetes.io/projected/211111e8-5f51-40fe-aaad-516c49d73212-kube-api-access-wdb88\") pod \"network-check-source-7d88cb55d8-5twc2\" (UID: \"211111e8-5f51-40fe-aaad-516c49d73212\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2" Jul 30 10:48:47.258471 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.258471 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.258376 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:47.758356843 +0000 UTC m=+35.731217857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : configmap references non-existent config key: service-ca.crt Jul 30 10:48:47.258471 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258401 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87f2m\" (UniqueName: \"kubernetes.io/projected/d78e2d15-951a-4df2-afce-ce74eb971837-kube-api-access-87f2m\") pod \"service-ca-operator-5d6c75f84f-f98ms\" (UID: \"d78e2d15-951a-4df2-afce-ce74eb971837\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.258471 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-default-certificate\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.258471 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.258451 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 10:48:47.258471 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hm48\" (UniqueName: \"kubernetes.io/projected/7d5c371a-680e-4c9b-aac7-aa5c13018096-kube-api-access-8hm48\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx\" (UID: \"7d5c371a-680e-4c9b-aac7-aa5c13018096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.258756 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bc5e620-0946-4c46-8fa7-c27d2ee33343-cert\") pod \"ingress-canary-b5vtk\" (UID: \"5bc5e620-0946-4c46-8fa7-c27d2ee33343\") " pod="openshift-ingress-canary/ingress-canary-b5vtk" Jul 30 10:48:47.258756 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.258508 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:47.758491273 +0000 UTC m=+35.731352301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : secret "router-metrics-certs-default" not found Jul 30 10:48:47.258756 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5c371a-680e-4c9b-aac7-aa5c13018096-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx\" (UID: \"7d5c371a-680e-4c9b-aac7-aa5c13018096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.258908 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258768 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7d5c371a-680e-4c9b-aac7-aa5c13018096-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx\" (UID: \"7d5c371a-680e-4c9b-aac7-aa5c13018096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.258908 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.258878 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d78e2d15-951a-4df2-afce-ce74eb971837-config\") pod \"service-ca-operator-5d6c75f84f-f98ms\" (UID: \"d78e2d15-951a-4df2-afce-ce74eb971837\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.260893 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.260869 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d78e2d15-951a-4df2-afce-ce74eb971837-serving-cert\") pod \"service-ca-operator-5d6c75f84f-f98ms\" (UID: \"d78e2d15-951a-4df2-afce-ce74eb971837\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.261224 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.261202 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5bc5e620-0946-4c46-8fa7-c27d2ee33343-cert\") pod \"ingress-canary-b5vtk\" (UID: \"5bc5e620-0946-4c46-8fa7-c27d2ee33343\") " pod="openshift-ingress-canary/ingress-canary-b5vtk" Jul 30 10:48:47.261329 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.261225 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-stats-auth\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.261329 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.261264 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-default-certificate\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.261329 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.261274 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7d5c371a-680e-4c9b-aac7-aa5c13018096-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx\" (UID: \"7d5c371a-680e-4c9b-aac7-aa5c13018096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.273891 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.273849 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hm48\" (UniqueName: \"kubernetes.io/projected/7d5c371a-680e-4c9b-aac7-aa5c13018096-kube-api-access-8hm48\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx\" (UID: \"7d5c371a-680e-4c9b-aac7-aa5c13018096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.274138 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.274115 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdb88\" (UniqueName: \"kubernetes.io/projected/211111e8-5f51-40fe-aaad-516c49d73212-kube-api-access-wdb88\") pod \"network-check-source-7d88cb55d8-5twc2\" (UID: \"211111e8-5f51-40fe-aaad-516c49d73212\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2" Jul 30 10:48:47.274357 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.274318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wtcj\" (UniqueName: \"kubernetes.io/projected/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-kube-api-access-6wtcj\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.274438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.274385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7xw\" (UniqueName: \"kubernetes.io/projected/5bc5e620-0946-4c46-8fa7-c27d2ee33343-kube-api-access-6d7xw\") pod \"ingress-canary-b5vtk\" (UID: \"5bc5e620-0946-4c46-8fa7-c27d2ee33343\") " pod="openshift-ingress-canary/ingress-canary-b5vtk" Jul 30 10:48:47.274494 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.274460 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-h8zf7" Jul 30 10:48:47.275077 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.275057 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87f2m\" (UniqueName: \"kubernetes.io/projected/d78e2d15-951a-4df2-afce-ce74eb971837-kube-api-access-87f2m\") pod \"service-ca-operator-5d6c75f84f-f98ms\" (UID: \"d78e2d15-951a-4df2-afce-ce74eb971837\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.275553 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.275529 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzvtq\" (UniqueName: \"kubernetes.io/projected/5acafbf4-35f9-49f6-abc9-5238956900fb-kube-api-access-wzvtq\") pod \"volume-data-source-validator-549c6d54cf-8kxd8\" (UID: \"5acafbf4-35f9-49f6-abc9-5238956900fb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8" Jul 30 10:48:47.325839 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.325804 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" Jul 30 10:48:47.336979 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.336942 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8" Jul 30 10:48:47.367543 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.367504 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2" Jul 30 10:48:47.378894 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.378854 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" Jul 30 10:48:47.393709 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.393676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b5vtk" Jul 30 10:48:47.521765 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.521672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:48:47.521765 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.521715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:48:47.521765 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.521715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:48:47.524384 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.524217 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 10:48:47.524384 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.524280 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gvflt\"" Jul 30 10:48:47.524384 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.524310 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwnnn\"" Jul 30 10:48:47.524384 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.524341 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 10:48:47.662476 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.662441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:47.662476 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.662481 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:47.662738 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.662525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:48:47.662738 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.662627 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 10:48:47.662738 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.662652 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcb89d8c9-shq24: secret "image-registry-tls" not found Jul 30 10:48:47.662738 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.662663 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 10:48:47.662738 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.662647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:47.662738 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.662709 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls podName:98410b9f-a736-487f-8c82-b3df27f5113b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:48.662690087 +0000 UTC m=+36.635551122 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls") pod "image-registry-7bcb89d8c9-shq24" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b") : secret "image-registry-tls" not found Jul 30 10:48:47.662738 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.662719 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 10:48:47.662738 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.662729 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls podName:fc7c9219-d5ac-4629-9b4f-0dcba70f47cd nodeName:}" failed. No retries permitted until 2026-07-30 10:48:48.66271997 +0000 UTC m=+36.635580986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls") pod "dns-default-msgck" (UID: "fc7c9219-d5ac-4629-9b4f-0dcba70f47cd") : secret "dns-default-metrics-tls" not found Jul 30 10:48:47.663154 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.662764 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls podName:9a6ca007-2d7e-45d8-861e-79c9b7d07655 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:48.662751793 +0000 UTC m=+36.635612816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-fz54r" (UID: "9a6ca007-2d7e-45d8-861e-79c9b7d07655") : secret "cluster-monitoring-operator-tls" not found Jul 30 10:48:47.663154 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.662821 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 10:48:47.663154 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.662858 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls podName:8c4dfaea-cd20-41df-90ea-0a961a08363d nodeName:}" failed. No retries permitted until 2026-07-30 10:48:48.662849294 +0000 UTC m=+36.635710307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls") pod "cluster-samples-operator-6748489656-m4mfj" (UID: "8c4dfaea-cd20-41df-90ea-0a961a08363d") : secret "samples-operator-tls" not found Jul 30 10:48:47.764039 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.763992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.764269 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:47.764117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:47.764269 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.764160 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 10:48:47.764269 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.764249 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:48.764220362 +0000 UTC m=+36.737081387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : secret "router-metrics-certs-default" not found Jul 30 10:48:47.764432 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:47.764273 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:48.764258103 +0000 UTC m=+36.737119116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : configmap references non-existent config key: service-ca.crt Jul 30 10:48:48.681377 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.680959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:48.681377 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.681303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:48.681377 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.681336 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:48.682116 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.681398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:48:48.682116 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.681157 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 10:48:48.682116 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.681587 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls podName:9a6ca007-2d7e-45d8-861e-79c9b7d07655 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:50.681567432 +0000 UTC m=+38.654428452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-fz54r" (UID: "9a6ca007-2d7e-45d8-861e-79c9b7d07655") : secret "cluster-monitoring-operator-tls" not found Jul 30 10:48:48.682116 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.681704 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 10:48:48.682116 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.681776 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls podName:fc7c9219-d5ac-4629-9b4f-0dcba70f47cd nodeName:}" failed. No retries permitted until 2026-07-30 10:48:50.681754647 +0000 UTC m=+38.654615683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls") pod "dns-default-msgck" (UID: "fc7c9219-d5ac-4629-9b4f-0dcba70f47cd") : secret "dns-default-metrics-tls" not found Jul 30 10:48:48.682116 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.681851 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 10:48:48.682116 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.681864 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcb89d8c9-shq24: secret "image-registry-tls" not found Jul 30 10:48:48.682116 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.681521 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 10:48:48.682116 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.681904 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls podName:98410b9f-a736-487f-8c82-b3df27f5113b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:50.681891013 +0000 UTC m=+38.654752050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls") pod "image-registry-7bcb89d8c9-shq24" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b") : secret "image-registry-tls" not found Jul 30 10:48:48.682116 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.681974 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls podName:8c4dfaea-cd20-41df-90ea-0a961a08363d nodeName:}" failed. No retries permitted until 2026-07-30 10:48:50.681941863 +0000 UTC m=+38.654802884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls") pod "cluster-samples-operator-6748489656-m4mfj" (UID: "8c4dfaea-cd20-41df-90ea-0a961a08363d") : secret "samples-operator-tls" not found Jul 30 10:48:48.716833 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.716759 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms"] Jul 30 10:48:48.724499 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.724469 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-h8zf7"] Jul 30 10:48:48.727932 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.727889 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2"] Jul 30 10:48:48.735570 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:48.735531 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78e2d15_951a_4df2_afce_ce74eb971837.slice/crio-2a11165339527baa8cfee43aacfe820db84bf1133c2b5ac3860676ef47f5fa8c WatchSource:0}: Error finding container 2a11165339527baa8cfee43aacfe820db84bf1133c2b5ac3860676ef47f5fa8c: Status 404 returned error can't find the container with id 2a11165339527baa8cfee43aacfe820db84bf1133c2b5ac3860676ef47f5fa8c Jul 30 10:48:48.735923 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:48.735903 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829982dc_21f7_4ee7_accb_0182ccd29add.slice/crio-0dadad92b1c1a29eda1501ca0605beefd2f100f290a6930cf8915706ede3e132 WatchSource:0}: Error finding container 0dadad92b1c1a29eda1501ca0605beefd2f100f290a6930cf8915706ede3e132: Status 404 returned error can't find the container with id 0dadad92b1c1a29eda1501ca0605beefd2f100f290a6930cf8915706ede3e132 Jul 30 10:48:48.737519 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:48.737356 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211111e8_5f51_40fe_aaad_516c49d73212.slice/crio-0bc5e81d7b9fe1ed1f8a657203d007d34103c2537530a7b8f992f75fdcb62706 WatchSource:0}: Error finding container 0bc5e81d7b9fe1ed1f8a657203d007d34103c2537530a7b8f992f75fdcb62706: Status 404 returned error can't find the container with id 0bc5e81d7b9fe1ed1f8a657203d007d34103c2537530a7b8f992f75fdcb62706 Jul 30 10:48:48.739660 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.739630 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8"] Jul 30 10:48:48.743493 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:48.743462 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acafbf4_35f9_49f6_abc9_5238956900fb.slice/crio-a35027b327d5974faf8a77b6595835fb5b956b157a032d6e148e931279f32149 WatchSource:0}: Error finding container a35027b327d5974faf8a77b6595835fb5b956b157a032d6e148e931279f32149: Status 404 returned error can't find the container with id a35027b327d5974faf8a77b6595835fb5b956b157a032d6e148e931279f32149 Jul 30 10:48:48.743712 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.743686 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mpcdj"] Jul 30 10:48:48.751274 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:48.751247 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8d2abb_c386_4bca_9278_c37aff6403fa.slice/crio-9b4a1e0eb64e0c60f1295919f32e64b5d5e0864b653505a538523d96447c8fda WatchSource:0}: Error finding container 9b4a1e0eb64e0c60f1295919f32e64b5d5e0864b653505a538523d96447c8fda: Status 404 returned error can't find the container with id 9b4a1e0eb64e0c60f1295919f32e64b5d5e0864b653505a538523d96447c8fda Jul 30 10:48:48.755274 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.755206 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b5vtk"] Jul 30 10:48:48.760552 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.760482 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx"] Jul 30 10:48:48.764597 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:48:48.764574 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5c371a_680e_4c9b_aac7_aa5c13018096.slice/crio-c71928419611a2359e6fec3c93286bcb9500001799d60a241cacac279d17f5ca WatchSource:0}: Error finding container c71928419611a2359e6fec3c93286bcb9500001799d60a241cacac279d17f5ca: Status 404 returned error can't find the container with id c71928419611a2359e6fec3c93286bcb9500001799d60a241cacac279d17f5ca Jul 30 10:48:48.782256 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.782235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:48.782393 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:48.782376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:48.782459 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.782401 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:50.78238347 +0000 UTC m=+38.755244483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : configmap references non-existent config key: service-ca.crt Jul 30 10:48:48.782519 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.782478 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 10:48:48.782564 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:48.782536 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:50.78251805 +0000 UTC m=+38.755379070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : secret "router-metrics-certs-default" not found Jul 30 10:48:49.696056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:49.696012 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" event={"ID":"d78e2d15-951a-4df2-afce-ce74eb971837","Type":"ContainerStarted","Data":"2a11165339527baa8cfee43aacfe820db84bf1133c2b5ac3860676ef47f5fa8c"} Jul 30 10:48:49.698760 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:49.698661 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2" event={"ID":"211111e8-5f51-40fe-aaad-516c49d73212","Type":"ContainerStarted","Data":"0bc5e81d7b9fe1ed1f8a657203d007d34103c2537530a7b8f992f75fdcb62706"} Jul 30 10:48:49.701783 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:49.701675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" event={"ID":"9b8d2abb-c386-4bca-9278-c37aff6403fa","Type":"ContainerStarted","Data":"9b4a1e0eb64e0c60f1295919f32e64b5d5e0864b653505a538523d96447c8fda"} Jul 30 10:48:49.705003 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:49.704966 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8" event={"ID":"5acafbf4-35f9-49f6-abc9-5238956900fb","Type":"ContainerStarted","Data":"a35027b327d5974faf8a77b6595835fb5b956b157a032d6e148e931279f32149"} Jul 30 10:48:49.706901 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:49.706829 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" event={"ID":"7d5c371a-680e-4c9b-aac7-aa5c13018096","Type":"ContainerStarted","Data":"c71928419611a2359e6fec3c93286bcb9500001799d60a241cacac279d17f5ca"} Jul 30 10:48:49.708746 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:49.708675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b5vtk" event={"ID":"5bc5e620-0946-4c46-8fa7-c27d2ee33343","Type":"ContainerStarted","Data":"a39237b259c6bd5cab5e010d1c32a091e5abcd0016e6063b13c6627f6c6d4b2b"} Jul 30 10:48:49.719236 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:49.718491 2575 generic.go:356] "Generic (PLEG): container finished" podID="125f3a8d-6f3a-434c-b8d7-45d9f56b32e5" containerID="fe7e5fab1a2cda7262b969dd34ee6dbb22639c1d128c8850b1814ddeb23ed59b" exitCode=0 Jul 30 10:48:49.719236 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:49.718591 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dspfd" event={"ID":"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5","Type":"ContainerDied","Data":"fe7e5fab1a2cda7262b969dd34ee6dbb22639c1d128c8850b1814ddeb23ed59b"} Jul 30 10:48:49.724036 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:49.723882 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-h8zf7" event={"ID":"829982dc-21f7-4ee7-accb-0182ccd29add","Type":"ContainerStarted","Data":"0dadad92b1c1a29eda1501ca0605beefd2f100f290a6930cf8915706ede3e132"} Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:50.704515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:50.704646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:50.704716 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:50.704747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.704852 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.704914 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.704929 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcb89d8c9-shq24: secret "image-registry-tls" not found Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.704918 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls podName:8c4dfaea-cd20-41df-90ea-0a961a08363d nodeName:}" failed. No retries permitted until 2026-07-30 10:48:54.704900802 +0000 UTC m=+42.677761816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls") pod "cluster-samples-operator-6748489656-m4mfj" (UID: "8c4dfaea-cd20-41df-90ea-0a961a08363d") : secret "samples-operator-tls" not found Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.704992 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls podName:98410b9f-a736-487f-8c82-b3df27f5113b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:54.704976847 +0000 UTC m=+42.677837866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls") pod "image-registry-7bcb89d8c9-shq24" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b") : secret "image-registry-tls" not found Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.705052 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.705083 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls podName:fc7c9219-d5ac-4629-9b4f-0dcba70f47cd nodeName:}" failed. No retries permitted until 2026-07-30 10:48:54.705072767 +0000 UTC m=+42.677933785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls") pod "dns-default-msgck" (UID: "fc7c9219-d5ac-4629-9b4f-0dcba70f47cd") : secret "dns-default-metrics-tls" not found Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.705149 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 10:48:50.705234 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.705181 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls podName:9a6ca007-2d7e-45d8-861e-79c9b7d07655 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:54.705171594 +0000 UTC m=+42.678032610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-fz54r" (UID: "9a6ca007-2d7e-45d8-861e-79c9b7d07655") : secret "cluster-monitoring-operator-tls" not found Jul 30 10:48:50.742872 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:50.741607 2575 generic.go:356] "Generic (PLEG): container finished" podID="125f3a8d-6f3a-434c-b8d7-45d9f56b32e5" containerID="6ec6b3b1eac47fd026150c404e902f8d159141be366c9636f9f7a9ccf4918ce7" exitCode=0 Jul 30 10:48:50.742872 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:50.741700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dspfd" event={"ID":"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5","Type":"ContainerDied","Data":"6ec6b3b1eac47fd026150c404e902f8d159141be366c9636f9f7a9ccf4918ce7"} Jul 30 10:48:50.805855 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:50.805814 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:50.806033 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:50.805921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:50.806152 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.806103 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:54.806067823 +0000 UTC m=+42.778928842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : configmap references non-existent config key: service-ca.crt Jul 30 10:48:50.806631 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.806507 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 10:48:50.806631 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:50.806562 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:48:54.806546859 +0000 UTC m=+42.779407879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : secret "router-metrics-certs-default" not found Jul 30 10:48:51.748459 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:51.748223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8" event={"ID":"5acafbf4-35f9-49f6-abc9-5238956900fb","Type":"ContainerStarted","Data":"35352346ff99943c9f5da0744f48e0af6a9d4fd25bb86b260d888399f0fe6ffa"} Jul 30 10:48:51.751769 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:51.751744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dspfd" event={"ID":"125f3a8d-6f3a-434c-b8d7-45d9f56b32e5","Type":"ContainerStarted","Data":"43b93e00a9161cb44cb9ef56d2c59a7e2dbfa992cedef48dc497f6ddd47b4b87"} Jul 30 10:48:51.765660 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:51.765607 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8kxd8" podStartSLOduration=11.531010506 podStartE2EDuration="13.765591059s" totalImagesPullingTime="2.234580553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:51.765379205 +0000 UTC m=+39.738240239" watchObservedRunningTime="2026-07-30 10:48:51.765591059 +0000 UTC m=+39.738452072" Jul 30 10:48:51.790894 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:51.790838 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dspfd" podStartSLOduration=9.093685824 podStartE2EDuration="39.790821036s" totalImagesPullingTime="30.697135212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:51.790540579 +0000 UTC m=+39.763401635" watchObservedRunningTime="2026-07-30 10:48:51.790821036 +0000 UTC m=+39.763682068" Jul 30 10:48:54.745757 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:54.745712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:54.745792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:54.745847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.745870 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.745916 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.745926 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcb89d8c9-shq24: secret "image-registry-tls" not found Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.745931 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls podName:8c4dfaea-cd20-41df-90ea-0a961a08363d nodeName:}" failed. No retries permitted until 2026-07-30 10:49:02.745916438 +0000 UTC m=+50.718777451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls") pod "cluster-samples-operator-6748489656-m4mfj" (UID: "8c4dfaea-cd20-41df-90ea-0a961a08363d") : secret "samples-operator-tls" not found Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:54.745950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.745962 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls podName:98410b9f-a736-487f-8c82-b3df27f5113b nodeName:}" failed. No retries permitted until 2026-07-30 10:49:02.745950505 +0000 UTC m=+50.718811522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls") pod "image-registry-7bcb89d8c9-shq24" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b") : secret "image-registry-tls" not found Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.745916 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.745996 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.746007 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls podName:9a6ca007-2d7e-45d8-861e-79c9b7d07655 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:02.745999125 +0000 UTC m=+50.718860138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-fz54r" (UID: "9a6ca007-2d7e-45d8-861e-79c9b7d07655") : secret "cluster-monitoring-operator-tls" not found Jul 30 10:48:54.746175 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.746022 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls podName:fc7c9219-d5ac-4629-9b4f-0dcba70f47cd nodeName:}" failed. No retries permitted until 2026-07-30 10:49:02.746013162 +0000 UTC m=+50.718874177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls") pod "dns-default-msgck" (UID: "fc7c9219-d5ac-4629-9b4f-0dcba70f47cd") : secret "dns-default-metrics-tls" not found Jul 30 10:48:54.847338 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:54.847296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:54.847505 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:54.847367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:48:54.847505 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.847433 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 10:48:54.847505 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.847481 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:49:02.847468904 +0000 UTC m=+50.820329921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : configmap references non-existent config key: service-ca.crt Jul 30 10:48:54.847505 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:54.847496 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:49:02.847489844 +0000 UTC m=+50.820350858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : secret "router-metrics-certs-default" not found Jul 30 10:48:56.775313 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.775113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" event={"ID":"7d5c371a-680e-4c9b-aac7-aa5c13018096","Type":"ContainerStarted","Data":"2b4cc9afc5c128e734eb4a933666ed4a4f5708aef666ff450082f8a59b930d7b"} Jul 30 10:48:56.777052 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.777025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b5vtk" event={"ID":"5bc5e620-0946-4c46-8fa7-c27d2ee33343","Type":"ContainerStarted","Data":"89365379dc3f493d2c4da20d5c4f131384abe958e69d93df031c00349537d2c5"} Jul 30 10:48:56.778529 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.778499 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-h8zf7" event={"ID":"829982dc-21f7-4ee7-accb-0182ccd29add","Type":"ContainerStarted","Data":"291d6b17191f5b5f101221316db2e1141bb2765f09ad1875caca5bbd94393349"} Jul 30 10:48:56.780220 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.780194 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" event={"ID":"d78e2d15-951a-4df2-afce-ce74eb971837","Type":"ContainerStarted","Data":"0700d77284ceb33de25e9709e3e32f4611387be791e1ee7f5653495e8f26baf8"} Jul 30 10:48:56.781896 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.781871 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2" event={"ID":"211111e8-5f51-40fe-aaad-516c49d73212","Type":"ContainerStarted","Data":"d9f8e8910e543f460db85a8d357237532baa7bb1b385ddc8503dfb21c2986e11"} Jul 30 10:48:56.783664 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.783645 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/0.log" Jul 30 10:48:56.783758 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.783682 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b8d2abb-c386-4bca-9278-c37aff6403fa" containerID="6f25611a389dc89d13166c7fafdc3b8dce338d913586759d4b596649b9b30ce1" exitCode=255 Jul 30 10:48:56.783758 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.783709 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" event={"ID":"9b8d2abb-c386-4bca-9278-c37aff6403fa","Type":"ContainerDied","Data":"6f25611a389dc89d13166c7fafdc3b8dce338d913586759d4b596649b9b30ce1"} Jul 30 10:48:56.783928 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.783913 2575 scope.go:117] "RemoveContainer" containerID="6f25611a389dc89d13166c7fafdc3b8dce338d913586759d4b596649b9b30ce1" Jul 30 10:48:56.793275 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.793230 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" podStartSLOduration=11.15173172 podStartE2EDuration="18.793213715s" totalImagesPullingTime="7.641481995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:56.792794183 +0000 UTC m=+44.765655218" watchObservedRunningTime="2026-07-30 10:48:56.793213715 +0000 UTC m=+44.766074749" Jul 30 10:48:56.832774 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.832542 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b5vtk" podStartSLOduration=3.182936276 podStartE2EDuration="10.832520658s" totalImagesPullingTime="7.649584382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:56.829868205 +0000 UTC m=+44.802729265" watchObservedRunningTime="2026-07-30 10:48:56.832520658 +0000 UTC m=+44.805381694" Jul 30 10:48:56.850937 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.850874 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-h8zf7" podStartSLOduration=12.187736362 podStartE2EDuration="19.850852271s" totalImagesPullingTime="7.663115909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:56.849948275 +0000 UTC m=+44.822809311" watchObservedRunningTime="2026-07-30 10:48:56.850852271 +0000 UTC m=+44.823713311" Jul 30 10:48:56.867990 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.867943 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" podStartSLOduration=11.20428335 podStartE2EDuration="18.867926813s" totalImagesPullingTime="7.663643463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:56.867190276 +0000 UTC m=+44.840051333" watchObservedRunningTime="2026-07-30 10:48:56.867926813 +0000 UTC m=+44.840787848" Jul 30 10:48:56.884765 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:56.884457 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-5twc2" podStartSLOduration=12.211730147 podStartE2EDuration="19.884439504s" totalImagesPullingTime="7.672709357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:48:56.884290213 +0000 UTC m=+44.857151249" watchObservedRunningTime="2026-07-30 10:48:56.884439504 +0000 UTC m=+44.857300537" Jul 30 10:48:57.246811 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.246770 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:57.247056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.246829 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:48:57.485961 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.485931 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j7wlw"] Jul 30 10:48:57.506055 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.505975 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7wlw"] Jul 30 10:48:57.506235 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.506120 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.508209 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.508165 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 10:48:57.508360 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.508251 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 10:48:57.508360 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.508348 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tdvpn\"" Jul 30 10:48:57.674102 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.674043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzwt\" (UniqueName: \"kubernetes.io/projected/af034d34-d609-4634-9780-261ddb318624-kube-api-access-rrzwt\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.674277 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.674168 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af034d34-d609-4634-9780-261ddb318624-crio-socket\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.674277 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.674213 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af034d34-d609-4634-9780-261ddb318624-data-volume\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.674277 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.674236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.674414 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.674387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.775755 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.775678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzwt\" (UniqueName: \"kubernetes.io/projected/af034d34-d609-4634-9780-261ddb318624-kube-api-access-rrzwt\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.775755 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.775742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af034d34-d609-4634-9780-261ddb318624-crio-socket\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.776148 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.775767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af034d34-d609-4634-9780-261ddb318624-data-volume\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.776148 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.775930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.776148 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.775981 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af034d34-d609-4634-9780-261ddb318624-crio-socket\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.776148 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:57.776044 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 10:48:57.776148 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.776061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af034d34-d609-4634-9780-261ddb318624-data-volume\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.776148 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.776082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.776148 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:57.776115 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm podName:af034d34-d609-4634-9780-261ddb318624 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:58.276082208 +0000 UTC m=+46.248943229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7wlw" (UID: "af034d34-d609-4634-9780-261ddb318624") : configmap "kube-rbac-proxy" not found Jul 30 10:48:57.776377 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:57.776169 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 10:48:57.776377 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:57.776227 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls podName:af034d34-d609-4634-9780-261ddb318624 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:58.27621069 +0000 UTC m=+46.249071711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7wlw" (UID: "af034d34-d609-4634-9780-261ddb318624") : secret "insights-runtime-extractor-tls" not found Jul 30 10:48:57.787994 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.787965 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzwt\" (UniqueName: \"kubernetes.io/projected/af034d34-d609-4634-9780-261ddb318624-kube-api-access-rrzwt\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:57.789517 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.789496 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/1.log" Jul 30 10:48:57.789947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.789930 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/0.log" Jul 30 10:48:57.790053 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.789968 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b8d2abb-c386-4bca-9278-c37aff6403fa" containerID="74a35adc185d65ad8c29b89a61f0f3f822b94608a3add6b5274631084e6292d6" exitCode=255 Jul 30 10:48:57.790129 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.790075 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" event={"ID":"9b8d2abb-c386-4bca-9278-c37aff6403fa","Type":"ContainerDied","Data":"74a35adc185d65ad8c29b89a61f0f3f822b94608a3add6b5274631084e6292d6"} Jul 30 10:48:57.790293 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.790167 2575 scope.go:117] "RemoveContainer" containerID="6f25611a389dc89d13166c7fafdc3b8dce338d913586759d4b596649b9b30ce1" Jul 30 10:48:57.790591 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:57.790548 2575 scope.go:117] "RemoveContainer" containerID="74a35adc185d65ad8c29b89a61f0f3f822b94608a3add6b5274631084e6292d6" Jul 30 10:48:57.790784 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:57.790745 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mpcdj_openshift-console-operator(9b8d2abb-c386-4bca-9278-c37aff6403fa)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" podUID="9b8d2abb-c386-4bca-9278-c37aff6403fa" Jul 30 10:48:58.281344 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:58.281308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:58.281528 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:58.281398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:58.281528 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:58.281449 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 10:48:58.281528 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:58.281498 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 10:48:58.281528 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:58.281518 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm podName:af034d34-d609-4634-9780-261ddb318624 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:59.281501355 +0000 UTC m=+47.254362367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7wlw" (UID: "af034d34-d609-4634-9780-261ddb318624") : configmap "kube-rbac-proxy" not found Jul 30 10:48:58.281676 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:58.281537 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls podName:af034d34-d609-4634-9780-261ddb318624 nodeName:}" failed. No retries permitted until 2026-07-30 10:48:59.281527048 +0000 UTC m=+47.254388061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7wlw" (UID: "af034d34-d609-4634-9780-261ddb318624") : secret "insights-runtime-extractor-tls" not found Jul 30 10:48:58.795154 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:58.795125 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/1.log" Jul 30 10:48:58.795637 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:58.795547 2575 scope.go:117] "RemoveContainer" containerID="74a35adc185d65ad8c29b89a61f0f3f822b94608a3add6b5274631084e6292d6" Jul 30 10:48:58.795774 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:58.795752 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mpcdj_openshift-console-operator(9b8d2abb-c386-4bca-9278-c37aff6403fa)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" podUID="9b8d2abb-c386-4bca-9278-c37aff6403fa" Jul 30 10:48:59.290189 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:59.290147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:59.290364 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:59.290256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:48:59.290364 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:59.290277 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 10:48:59.290364 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:59.290351 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm podName:af034d34-d609-4634-9780-261ddb318624 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:01.290331267 +0000 UTC m=+49.263192299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7wlw" (UID: "af034d34-d609-4634-9780-261ddb318624") : configmap "kube-rbac-proxy" not found Jul 30 10:48:59.290485 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:59.290377 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 10:48:59.290485 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:48:59.290413 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls podName:af034d34-d609-4634-9780-261ddb318624 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:01.290402524 +0000 UTC m=+49.263263548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7wlw" (UID: "af034d34-d609-4634-9780-261ddb318624") : secret "insights-runtime-extractor-tls" not found Jul 30 10:48:59.295000 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:48:59.294979 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5vrl_f8307c4d-a9aa-4e53-9b75-ca7bc4080028/dns-node-resolver/0.log" Jul 30 10:49:00.495923 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:00.495892 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qwpkd_177e8448-e3d4-40b5-aa14-9889d7901b64/node-ca/0.log" Jul 30 10:49:01.095656 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:01.095617 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5vtk_5bc5e620-0946-4c46-8fa7-c27d2ee33343/serve-healthcheck-canary/0.log" Jul 30 10:49:01.307715 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:01.307662 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:49:01.307923 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:01.307772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:49:01.307923 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:01.307826 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 10:49:01.307923 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:01.307855 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 10:49:01.307923 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:01.307899 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm podName:af034d34-d609-4634-9780-261ddb318624 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:05.307884683 +0000 UTC m=+53.280745697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm") pod "insights-runtime-extractor-j7wlw" (UID: "af034d34-d609-4634-9780-261ddb318624") : configmap "kube-rbac-proxy" not found Jul 30 10:49:01.307923 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:01.307911 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls podName:af034d34-d609-4634-9780-261ddb318624 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:05.30790561 +0000 UTC m=+53.280766623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7wlw" (UID: "af034d34-d609-4634-9780-261ddb318624") : secret "insights-runtime-extractor-tls" not found Jul 30 10:49:02.821397 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:02.821357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:02.821427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:02.821448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:02.821486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.821508 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.821570 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls podName:9a6ca007-2d7e-45d8-861e-79c9b7d07655 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:18.821552988 +0000 UTC m=+66.794414000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-fz54r" (UID: "9a6ca007-2d7e-45d8-861e-79c9b7d07655") : secret "cluster-monitoring-operator-tls" not found Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.821595 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.821604 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.821622 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.821636 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls podName:8c4dfaea-cd20-41df-90ea-0a961a08363d nodeName:}" failed. No retries permitted until 2026-07-30 10:49:18.82162443 +0000 UTC m=+66.794485448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls") pod "cluster-samples-operator-6748489656-m4mfj" (UID: "8c4dfaea-cd20-41df-90ea-0a961a08363d") : secret "samples-operator-tls" not found Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.821638 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcb89d8c9-shq24: secret "image-registry-tls" not found Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.821656 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls podName:fc7c9219-d5ac-4629-9b4f-0dcba70f47cd nodeName:}" failed. No retries permitted until 2026-07-30 10:49:18.82164148 +0000 UTC m=+66.794502500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls") pod "dns-default-msgck" (UID: "fc7c9219-d5ac-4629-9b4f-0dcba70f47cd") : secret "dns-default-metrics-tls" not found Jul 30 10:49:02.821862 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.821682 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls podName:98410b9f-a736-487f-8c82-b3df27f5113b nodeName:}" failed. No retries permitted until 2026-07-30 10:49:18.821668704 +0000 UTC m=+66.794529720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls") pod "image-registry-7bcb89d8c9-shq24" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b") : secret "image-registry-tls" not found Jul 30 10:49:02.922600 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:02.922562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:02.922775 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.922713 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 30 10:49:02.922775 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:02.922742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:02.922775 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.922772 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:49:18.922757822 +0000 UTC m=+66.895618834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : secret "router-metrics-certs-default" not found Jul 30 10:49:02.923000 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:02.922851 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle podName:5f89b351-afdb-4b3e-8e8d-94f663c1c65b nodeName:}" failed. No retries permitted until 2026-07-30 10:49:18.922835808 +0000 UTC m=+66.895696824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle") pod "router-default-5984b4bcb7-nxpbf" (UID: "5f89b351-afdb-4b3e-8e8d-94f663c1c65b") : configmap references non-existent config key: service-ca.crt Jul 30 10:49:05.346076 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:05.346016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:49:05.346542 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:05.346153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:49:05.346542 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:05.346315 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 10:49:05.346542 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:05.346400 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls podName:af034d34-d609-4634-9780-261ddb318624 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:13.346378712 +0000 UTC m=+61.319239734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls") pod "insights-runtime-extractor-j7wlw" (UID: "af034d34-d609-4634-9780-261ddb318624") : secret "insights-runtime-extractor-tls" not found Jul 30 10:49:05.346686 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:05.346664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af034d34-d609-4634-9780-261ddb318624-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:49:07.247188 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:07.247147 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:49:07.247578 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:07.247205 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:49:07.247578 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:07.247549 2575 scope.go:117] "RemoveContainer" containerID="74a35adc185d65ad8c29b89a61f0f3f822b94608a3add6b5274631084e6292d6" Jul 30 10:49:07.825186 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:07.825159 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 10:49:07.825586 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:07.825572 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/1.log" Jul 30 10:49:07.825639 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:07.825604 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b8d2abb-c386-4bca-9278-c37aff6403fa" containerID="7cb987ca3f35c748f561058afaaa8b63bdc3ed078b85c066e20fa3401c060ccf" exitCode=255 Jul 30 10:49:07.825681 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:07.825651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" event={"ID":"9b8d2abb-c386-4bca-9278-c37aff6403fa","Type":"ContainerDied","Data":"7cb987ca3f35c748f561058afaaa8b63bdc3ed078b85c066e20fa3401c060ccf"} Jul 30 10:49:07.825734 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:07.825697 2575 scope.go:117] "RemoveContainer" containerID="74a35adc185d65ad8c29b89a61f0f3f822b94608a3add6b5274631084e6292d6" Jul 30 10:49:07.826027 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:07.826010 2575 scope.go:117] "RemoveContainer" containerID="7cb987ca3f35c748f561058afaaa8b63bdc3ed078b85c066e20fa3401c060ccf" Jul 30 10:49:07.826245 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:07.826225 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-mpcdj_openshift-console-operator(9b8d2abb-c386-4bca-9278-c37aff6403fa)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" podUID="9b8d2abb-c386-4bca-9278-c37aff6403fa" Jul 30 10:49:08.829907 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:08.829878 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 10:49:12.681178 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:12.681146 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2z7n2" Jul 30 10:49:13.420709 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:13.420674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:49:13.422983 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:13.422957 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af034d34-d609-4634-9780-261ddb318624-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wlw\" (UID: \"af034d34-d609-4634-9780-261ddb318624\") " pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:49:13.718519 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:13.718438 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tdvpn\"" Jul 30 10:49:13.727032 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:13.727012 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7wlw" Jul 30 10:49:13.850930 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:13.850907 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7wlw"] Jul 30 10:49:13.853464 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:13.853441 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf034d34_d609_4634_9780_261ddb318624.slice/crio-27eb2d246d9fc581f5805412b953c635249604cc0d7054be45e139b79967a568 WatchSource:0}: Error finding container 27eb2d246d9fc581f5805412b953c635249604cc0d7054be45e139b79967a568: Status 404 returned error can't find the container with id 27eb2d246d9fc581f5805412b953c635249604cc0d7054be45e139b79967a568 Jul 30 10:49:14.849160 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:14.849116 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7wlw" event={"ID":"af034d34-d609-4634-9780-261ddb318624","Type":"ContainerStarted","Data":"27eb2d246d9fc581f5805412b953c635249604cc0d7054be45e139b79967a568"} Jul 30 10:49:17.247361 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:17.247314 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:49:17.247799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:17.247391 2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:49:17.247799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:17.247753 2575 scope.go:117] "RemoveContainer" containerID="7cb987ca3f35c748f561058afaaa8b63bdc3ed078b85c066e20fa3401c060ccf" Jul 30 10:49:17.248001 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:17.247977 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-mpcdj_openshift-console-operator(9b8d2abb-c386-4bca-9278-c37aff6403fa)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" podUID="9b8d2abb-c386-4bca-9278-c37aff6403fa" Jul 30 10:49:18.167879 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.167826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:49:18.170381 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.170355 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 10:49:18.181258 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.181225 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f31dbccc-5060-4cb2-9df5-279124fcd806-original-pull-secret\") pod \"global-pull-secret-syncer-452jf\" (UID: \"f31dbccc-5060-4cb2-9df5-279124fcd806\") " pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:49:18.268956 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.268915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:49:18.271807 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.271784 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 10:49:18.281821 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.281791 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb-metrics-certs\") pod \"network-metrics-daemon-chjhd\" (UID: \"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb\") " pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:49:18.369577 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.369537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8pc\" (UniqueName: \"kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc\") pod \"network-check-target-cdb5g\" (UID: \"d86e960e-cae8-4510-afcd-37dc31b105e6\") " pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:49:18.372212 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.372182 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8pc\" (UniqueName: \"kubernetes.io/projected/d86e960e-cae8-4510-afcd-37dc31b105e6-kube-api-access-hb8pc\") pod \"network-check-target-cdb5g\" (UID: \"d86e960e-cae8-4510-afcd-37dc31b105e6\") " pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:49:18.436968 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.436889 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwnnn\"" Jul 30 10:49:18.444716 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.444681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:49:18.444871 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.444758 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gvflt\"" Jul 30 10:49:18.449484 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.449463 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-452jf" Jul 30 10:49:18.453599 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.453581 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-chjhd" Jul 30 10:49:18.874165 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.874129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:49:18.874337 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.874192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:49:18.874337 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.874214 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:49:18.874418 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.874346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:49:18.876720 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.876695 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") pod \"image-registry-7bcb89d8c9-shq24\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:49:18.876720 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.876701 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a6ca007-2d7e-45d8-861e-79c9b7d07655-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-fz54r\" (UID: \"9a6ca007-2d7e-45d8-861e-79c9b7d07655\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:49:18.876911 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.876793 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c4dfaea-cd20-41df-90ea-0a961a08363d-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-m4mfj\" (UID: \"8c4dfaea-cd20-41df-90ea-0a961a08363d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:49:18.876949 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.876928 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc7c9219-d5ac-4629-9b4f-0dcba70f47cd-metrics-tls\") pod \"dns-default-msgck\" (UID: \"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd\") " pod="openshift-dns/dns-default-msgck" Jul 30 10:49:18.975140 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.975074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:18.975346 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.975202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:18.975828 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.975803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-service-ca-bundle\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:18.977887 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:18.977861 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f89b351-afdb-4b3e-8e8d-94f663c1c65b-metrics-certs\") pod \"router-default-5984b4bcb7-nxpbf\" (UID: \"5f89b351-afdb-4b3e-8e8d-94f663c1c65b\") " pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:19.051401 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.051370 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-g9vp6\"" Jul 30 10:49:19.059160 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.059128 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" Jul 30 10:49:19.088471 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.088442 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpkjw\"" Jul 30 10:49:19.096167 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.096137 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:49:19.099994 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.099970 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vdjzf\"" Jul 30 10:49:19.108122 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.108082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" Jul 30 10:49:19.114923 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.114902 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6l9vk\"" Jul 30 10:49:19.122968 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.122942 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-msgck" Jul 30 10:49:19.150416 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.150390 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-45qtj\"" Jul 30 10:49:19.159210 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.159168 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:19.533730 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.533556 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bcb89d8c9-shq24"] Jul 30 10:49:19.536128 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:19.536052 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98410b9f_a736_487f_8c82_b3df27f5113b.slice/crio-2314539a2d81e1694e23f689d91c705fe6b0ddc942bcda37d8f1a2d0949af2a8 WatchSource:0}: Error finding container 2314539a2d81e1694e23f689d91c705fe6b0ddc942bcda37d8f1a2d0949af2a8: Status 404 returned error can't find the container with id 2314539a2d81e1694e23f689d91c705fe6b0ddc942bcda37d8f1a2d0949af2a8 Jul 30 10:49:19.551082 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.551045 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5984b4bcb7-nxpbf"] Jul 30 10:49:19.553892 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:19.553845 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f89b351_afdb_4b3e_8e8d_94f663c1c65b.slice/crio-45e643f86cdafad05660bf27f8398660d1f7990a0b2cd073293ebe5c0ab43728 WatchSource:0}: Error finding container 45e643f86cdafad05660bf27f8398660d1f7990a0b2cd073293ebe5c0ab43728: Status 404 returned error can't find the container with id 45e643f86cdafad05660bf27f8398660d1f7990a0b2cd073293ebe5c0ab43728 Jul 30 10:49:19.771367 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.771288 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cdb5g"] Jul 30 10:49:19.775152 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.774808 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r"] Jul 30 10:49:19.776151 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:19.776049 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86e960e_cae8_4510_afcd_37dc31b105e6.slice/crio-90aca5c917bda4b963406ffa48d9594eb4afe15fcb8ce74005e17390fe82a65a WatchSource:0}: Error finding container 90aca5c917bda4b963406ffa48d9594eb4afe15fcb8ce74005e17390fe82a65a: Status 404 returned error can't find the container with id 90aca5c917bda4b963406ffa48d9594eb4afe15fcb8ce74005e17390fe82a65a Jul 30 10:49:19.776151 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.776060 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-452jf"] Jul 30 10:49:19.778259 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:19.778205 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31dbccc_5060_4cb2_9df5_279124fcd806.slice/crio-3a37372adc0a380a800db4d0c97f9a4ad3e05e5444149fc84d69ce9213bb6ccd WatchSource:0}: Error finding container 3a37372adc0a380a800db4d0c97f9a4ad3e05e5444149fc84d69ce9213bb6ccd: Status 404 returned error can't find the container with id 3a37372adc0a380a800db4d0c97f9a4ad3e05e5444149fc84d69ce9213bb6ccd Jul 30 10:49:19.779434 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:19.779055 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6ca007_2d7e_45d8_861e_79c9b7d07655.slice/crio-fdcee8de0a90aabfb746052b8514a2896fe2c15280fa76a511e6dbe9823d5b44 WatchSource:0}: Error finding container fdcee8de0a90aabfb746052b8514a2896fe2c15280fa76a511e6dbe9823d5b44: Status 404 returned error can't find the container with id fdcee8de0a90aabfb746052b8514a2896fe2c15280fa76a511e6dbe9823d5b44 Jul 30 10:49:19.824386 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.824345 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj"] Jul 30 10:49:19.826777 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.826599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-msgck"] Jul 30 10:49:19.830340 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.830268 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-chjhd"] Jul 30 10:49:19.830755 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:19.830708 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7c9219_d5ac_4629_9b4f_0dcba70f47cd.slice/crio-1b1eff9bf8a101f0592047c43a4437db42870174b8270cb706db92db2729fbe7 WatchSource:0}: Error finding container 1b1eff9bf8a101f0592047c43a4437db42870174b8270cb706db92db2729fbe7: Status 404 returned error can't find the container with id 1b1eff9bf8a101f0592047c43a4437db42870174b8270cb706db92db2729fbe7 Jul 30 10:49:19.835302 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:19.835268 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50d2e2f_3f1b_4f00_9b34_bd2f23e02dfb.slice/crio-50e44cbc534ccbf27d302cc2913c7afaf8d245b3fc6842bf4e72bc2eea8188f3 WatchSource:0}: Error finding container 50e44cbc534ccbf27d302cc2913c7afaf8d245b3fc6842bf4e72bc2eea8188f3: Status 404 returned error can't find the container with id 50e44cbc534ccbf27d302cc2913c7afaf8d245b3fc6842bf4e72bc2eea8188f3 Jul 30 10:49:19.871440 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.871403 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-chjhd" event={"ID":"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb","Type":"ContainerStarted","Data":"50e44cbc534ccbf27d302cc2913c7afaf8d245b3fc6842bf4e72bc2eea8188f3"} Jul 30 10:49:19.872611 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.872586 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" event={"ID":"8c4dfaea-cd20-41df-90ea-0a961a08363d","Type":"ContainerStarted","Data":"632f4d633cea795989adcf8553031c4218ea9b5cc0bdf770e776f8c1a1ae80f1"} Jul 30 10:49:19.873762 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.873743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" event={"ID":"9a6ca007-2d7e-45d8-861e-79c9b7d07655","Type":"ContainerStarted","Data":"fdcee8de0a90aabfb746052b8514a2896fe2c15280fa76a511e6dbe9823d5b44"} Jul 30 10:49:19.874877 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.874856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-msgck" event={"ID":"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd","Type":"ContainerStarted","Data":"1b1eff9bf8a101f0592047c43a4437db42870174b8270cb706db92db2729fbe7"} Jul 30 10:49:19.876504 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.876485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cdb5g" event={"ID":"d86e960e-cae8-4510-afcd-37dc31b105e6","Type":"ContainerStarted","Data":"3cd303cd82f0eac5a747f4fa25318962f082eead63a745a4b06e911a93ce71a8"} Jul 30 10:49:19.876604 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.876523 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cdb5g" event={"ID":"d86e960e-cae8-4510-afcd-37dc31b105e6","Type":"ContainerStarted","Data":"90aca5c917bda4b963406ffa48d9594eb4afe15fcb8ce74005e17390fe82a65a"} Jul 30 10:49:19.876604 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.876579 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:49:19.877747 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.877717 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-452jf" event={"ID":"f31dbccc-5060-4cb2-9df5-279124fcd806","Type":"ContainerStarted","Data":"3a37372adc0a380a800db4d0c97f9a4ad3e05e5444149fc84d69ce9213bb6ccd"} Jul 30 10:49:19.879206 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.879179 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7wlw" event={"ID":"af034d34-d609-4634-9780-261ddb318624","Type":"ContainerStarted","Data":"198233c555c554aab5e0419b8e13b40eb4167618c8ea6f7a616902e617bc4628"} Jul 30 10:49:19.880722 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.880702 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" event={"ID":"5f89b351-afdb-4b3e-8e8d-94f663c1c65b","Type":"ContainerStarted","Data":"72dc0156773877e97838b1f9d85dc75d4f1f3ed44a38bd41c563e77905d4bb63"} Jul 30 10:49:19.880802 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.880739 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" event={"ID":"5f89b351-afdb-4b3e-8e8d-94f663c1c65b","Type":"ContainerStarted","Data":"45e643f86cdafad05660bf27f8398660d1f7990a0b2cd073293ebe5c0ab43728"} Jul 30 10:49:19.882398 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.882379 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" event={"ID":"98410b9f-a736-487f-8c82-b3df27f5113b","Type":"ContainerStarted","Data":"748e5c41fd96a2b8cc9b6f3703d65cd2177bce19329f6c4da158e1bbb4ba86cc"} Jul 30 10:49:19.882481 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.882413 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" event={"ID":"98410b9f-a736-487f-8c82-b3df27f5113b","Type":"ContainerStarted","Data":"2314539a2d81e1694e23f689d91c705fe6b0ddc942bcda37d8f1a2d0949af2a8"} Jul 30 10:49:19.882529 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.882484 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:49:19.893482 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.893424 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cdb5g" podStartSLOduration=67.893405502 podStartE2EDuration="1m7.893405502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:19.892753137 +0000 UTC m=+67.865614174" watchObservedRunningTime="2026-07-30 10:49:19.893405502 +0000 UTC m=+67.866266538" Jul 30 10:49:19.918105 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.918029 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" podStartSLOduration=41.9180112 podStartE2EDuration="41.9180112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:19.91661596 +0000 UTC m=+67.889476998" watchObservedRunningTime="2026-07-30 10:49:19.9180112 +0000 UTC m=+67.890872235" Jul 30 10:49:19.941458 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:19.941259 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" podStartSLOduration=67.941233958 podStartE2EDuration="1m7.941233958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:19.940741971 +0000 UTC m=+67.913603006" watchObservedRunningTime="2026-07-30 10:49:19.941233958 +0000 UTC m=+67.914094991" Jul 30 10:49:20.159483 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:20.159407 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:20.162405 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:20.162268 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:20.890850 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:20.890782 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:20.894041 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:20.893839 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5984b4bcb7-nxpbf" Jul 30 10:49:21.893704 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:21.893661 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-chjhd" event={"ID":"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb","Type":"ContainerStarted","Data":"3a3d98d644bfe68c729ea3d9b5ac7f9e661033bd7595df457729653f0831537a"} Jul 30 10:49:21.894196 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:21.893760 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-chjhd" event={"ID":"d50d2e2f-3f1b-4f00-9b34-bd2f23e02dfb","Type":"ContainerStarted","Data":"d33ba0722d653ec4ff46bea9a56e1c14e45e65f1d98bc24529ce10d6e9a40097"} Jul 30 10:49:21.910804 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:21.910746 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-chjhd" podStartSLOduration=68.572185902 podStartE2EDuration="1m9.910721785s" totalImagesPullingTime="1.338535883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:21.91006169 +0000 UTC m=+69.882922726" watchObservedRunningTime="2026-07-30 10:49:21.910721785 +0000 UTC m=+69.883582821" Jul 30 10:49:27.925381 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.925338 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" event={"ID":"8c4dfaea-cd20-41df-90ea-0a961a08363d","Type":"ContainerStarted","Data":"71abf5092dfa47d91d53f3e093861c08098fac3ebaa9a74562a878fd568468cd"} Jul 30 10:49:27.925857 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.925418 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" event={"ID":"8c4dfaea-cd20-41df-90ea-0a961a08363d","Type":"ContainerStarted","Data":"7bb9c93d1e711e23a3552d20202be729d2366de8fcdb6f49170600bab1cbcebb"} Jul 30 10:49:27.927423 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.927361 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" event={"ID":"9a6ca007-2d7e-45d8-861e-79c9b7d07655","Type":"ContainerStarted","Data":"e839ed1dc4d03b82e2ca0214ce673c614bfb59c9f6d39da61517b738bbda34f3"} Jul 30 10:49:27.929545 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.929492 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-msgck" event={"ID":"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd","Type":"ContainerStarted","Data":"8659671a457c063e528340c7095546985bcb544e2b8b9259e1a2d7a5c1b0fcf5"} Jul 30 10:49:27.929545 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.929539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-msgck" event={"ID":"fc7c9219-d5ac-4629-9b4f-0dcba70f47cd","Type":"ContainerStarted","Data":"382eb2e4204c220e08c4b6eb855a7a4b1f693bd3c8cc2e8d9e8f3bcaa7bc3fe5"} Jul 30 10:49:27.929744 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.929569 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-msgck" Jul 30 10:49:27.931070 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.931044 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-452jf" event={"ID":"f31dbccc-5060-4cb2-9df5-279124fcd806","Type":"ContainerStarted","Data":"303e5ad16d0a1a2682c8fd01942d164feb577e29477c919796fedd7fac1e5a0d"} Jul 30 10:49:27.932961 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.932939 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7wlw" event={"ID":"af034d34-d609-4634-9780-261ddb318624","Type":"ContainerStarted","Data":"ee399cdce0dc9ee6c122dac674920ba9d38de1ca16eb7c4f992117852409118c"} Jul 30 10:49:27.945329 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.945284 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-m4mfj" podStartSLOduration=44.065671482 podStartE2EDuration="50.94526861s" totalImagesPullingTime="6.879597128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:27.944394974 +0000 UTC m=+75.917256036" watchObservedRunningTime="2026-07-30 10:49:27.94526861 +0000 UTC m=+75.918129644" Jul 30 10:49:27.959562 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.959511 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-452jf" podStartSLOduration=68.704202811 podStartE2EDuration="1m15.959492903s" totalImagesPullingTime="7.255290092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:27.958813855 +0000 UTC m=+75.931674910" watchObservedRunningTime="2026-07-30 10:49:27.959492903 +0000 UTC m=+75.932353942" Jul 30 10:49:27.979050 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.978995 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-msgck" podStartSLOduration=35.065684528 podStartE2EDuration="41.978974073s" totalImagesPullingTime="6.913289545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:27.97818605 +0000 UTC m=+75.951047086" watchObservedRunningTime="2026-07-30 10:49:27.978974073 +0000 UTC m=+75.951835109" Jul 30 10:49:27.995525 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:27.995470 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-fz54r" podStartSLOduration=44.029789503 podStartE2EDuration="50.995448181s" totalImagesPullingTime="6.965658678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:27.993588721 +0000 UTC m=+75.966449768" watchObservedRunningTime="2026-07-30 10:49:27.995448181 +0000 UTC m=+75.968309217" Jul 30 10:49:28.521412 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:28.521371 2575 scope.go:117] "RemoveContainer" containerID="7cb987ca3f35c748f561058afaaa8b63bdc3ed078b85c066e20fa3401c060ccf" Jul 30 10:49:28.939262 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:28.939225 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 10:49:28.939748 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:28.939393 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" event={"ID":"9b8d2abb-c386-4bca-9278-c37aff6403fa","Type":"ContainerStarted","Data":"838e84da49e77951664f164a5beac615de6514eebe1d06462f4250dd96a909f5"} Jul 30 10:49:28.940042 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:28.940012 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:49:28.958123 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:28.958052 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" podStartSLOduration=44.30277765 podStartE2EDuration="51.95803098s" totalImagesPullingTime="7.65525333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:28.956644209 +0000 UTC m=+76.929505269" watchObservedRunningTime="2026-07-30 10:49:28.95803098 +0000 UTC m=+76.930892016" Jul 30 10:49:29.350056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:29.350023 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-mpcdj" Jul 30 10:49:31.953157 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:31.953118 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7wlw" event={"ID":"af034d34-d609-4634-9780-261ddb318624","Type":"ContainerStarted","Data":"cafa862fba921f4be41c2e34ce4ddff0f6d6b0151072fe7065eba64a7b2772f2"} Jul 30 10:49:31.973924 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:31.973865 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j7wlw" podStartSLOduration=17.9667165 podStartE2EDuration="34.973845961s" totalImagesPullingTime="17.007129461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 10:48:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:31.972456392 +0000 UTC m=+79.945317427" watchObservedRunningTime="2026-07-30 10:49:31.973845961 +0000 UTC m=+79.946706996" Jul 30 10:49:33.747371 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.747216 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-48hr7"] Jul 30 10:49:33.750269 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.750246 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:33.753174 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.753148 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 30 10:49:33.753393 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.753379 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 30 10:49:33.753749 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.753730 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-c4td8\"" Jul 30 10:49:33.754101 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.754064 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 10:49:33.769949 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.769905 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-48hr7"] Jul 30 10:49:33.904635 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.904592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc07d297-c953-45b8-81c5-f187d601e331-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:33.904635 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.904632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc07d297-c953-45b8-81c5-f187d601e331-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:33.904892 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.904669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc07d297-c953-45b8-81c5-f187d601e331-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:33.904892 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:33.904794 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9846t\" (UniqueName: \"kubernetes.io/projected/cc07d297-c953-45b8-81c5-f187d601e331-kube-api-access-9846t\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.006047 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.005955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc07d297-c953-45b8-81c5-f187d601e331-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.006047 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.005993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc07d297-c953-45b8-81c5-f187d601e331-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.006047 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.006020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc07d297-c953-45b8-81c5-f187d601e331-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.006367 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.006067 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9846t\" (UniqueName: \"kubernetes.io/projected/cc07d297-c953-45b8-81c5-f187d601e331-kube-api-access-9846t\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.006367 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:34.006150 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 30 10:49:34.006367 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:49:34.006217 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc07d297-c953-45b8-81c5-f187d601e331-prometheus-operator-tls podName:cc07d297-c953-45b8-81c5-f187d601e331 nodeName:}" failed. No retries permitted until 2026-07-30 10:49:34.50619931 +0000 UTC m=+82.479060327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/cc07d297-c953-45b8-81c5-f187d601e331-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-48hr7" (UID: "cc07d297-c953-45b8-81c5-f187d601e331") : secret "prometheus-operator-tls" not found Jul 30 10:49:34.006696 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.006676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc07d297-c953-45b8-81c5-f187d601e331-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.008430 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.008410 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc07d297-c953-45b8-81c5-f187d601e331-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.020619 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.020577 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9846t\" (UniqueName: \"kubernetes.io/projected/cc07d297-c953-45b8-81c5-f187d601e331-kube-api-access-9846t\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.510549 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.510512 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc07d297-c953-45b8-81c5-f187d601e331-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.512912 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.512889 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc07d297-c953-45b8-81c5-f187d601e331-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-48hr7\" (UID: \"cc07d297-c953-45b8-81c5-f187d601e331\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.666985 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.666954 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" Jul 30 10:49:34.805349 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.805316 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-48hr7"] Jul 30 10:49:34.808330 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:34.808288 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc07d297_c953_45b8_81c5_f187d601e331.slice/crio-d1430b8a558c71fc008149cf9506ed87501c5412a9c36cd3fd6a43f3a7572c59 WatchSource:0}: Error finding container d1430b8a558c71fc008149cf9506ed87501c5412a9c36cd3fd6a43f3a7572c59: Status 404 returned error can't find the container with id d1430b8a558c71fc008149cf9506ed87501c5412a9c36cd3fd6a43f3a7572c59 Jul 30 10:49:34.967151 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:34.967118 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" event={"ID":"cc07d297-c953-45b8-81c5-f187d601e331","Type":"ContainerStarted","Data":"d1430b8a558c71fc008149cf9506ed87501c5412a9c36cd3fd6a43f3a7572c59"} Jul 30 10:49:37.942127 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:37.942015 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-msgck" Jul 30 10:49:37.981899 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:37.981867 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" event={"ID":"cc07d297-c953-45b8-81c5-f187d601e331","Type":"ContainerStarted","Data":"4ed9bcd8cb07c6e55c2e4b2784266acb16494a09add471b6515fb2f0949f2544"} Jul 30 10:49:37.982083 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:37.981926 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" event={"ID":"cc07d297-c953-45b8-81c5-f187d601e331","Type":"ContainerStarted","Data":"ffdfdcce91a6b79642f4ba7211368753b1cc76fd753742acd857e7073c258f50"} Jul 30 10:49:38.003312 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:38.003236 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-48hr7" podStartSLOduration=2.188339828 podStartE2EDuration="5.003214148s" totalImagesPullingTime="2.81487432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:37.999291948 +0000 UTC m=+85.972153004" watchObservedRunningTime="2026-07-30 10:49:38.003214148 +0000 UTC m=+85.976075183" Jul 30 10:49:40.129816 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.129778 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-svgfs"] Jul 30 10:49:40.132681 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.132665 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.135317 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.135291 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 10:49:40.135445 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.135349 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 10:49:40.135445 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.135362 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-r6g6q\"" Jul 30 10:49:40.135445 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.135374 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 10:49:40.255882 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.255839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfdce1f4-44e6-4e54-84c1-eb7f45512086-sys\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.255882 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.255882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-accelerators-collector-config\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.256202 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.255906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-textfile\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.256202 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.255998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-wtmp\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.256202 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.256045 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.256202 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.256121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfdce1f4-44e6-4e54-84c1-eb7f45512086-metrics-client-ca\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.256202 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.256183 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqwj\" (UniqueName: \"kubernetes.io/projected/cfdce1f4-44e6-4e54-84c1-eb7f45512086-kube-api-access-5fqwj\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.256440 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.256247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-tls\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.256440 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.256299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cfdce1f4-44e6-4e54-84c1-eb7f45512086-root\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357196 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfdce1f4-44e6-4e54-84c1-eb7f45512086-sys\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357196 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-accelerators-collector-config\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-textfile\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-wtmp\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfdce1f4-44e6-4e54-84c1-eb7f45512086-metrics-client-ca\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfdce1f4-44e6-4e54-84c1-eb7f45512086-sys\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357309 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqwj\" (UniqueName: \"kubernetes.io/projected/cfdce1f4-44e6-4e54-84c1-eb7f45512086-kube-api-access-5fqwj\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357335 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-tls\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cfdce1f4-44e6-4e54-84c1-eb7f45512086-root\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357818 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357439 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-wtmp\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357818 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cfdce1f4-44e6-4e54-84c1-eb7f45512086-root\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357818 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357607 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-textfile\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357933 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-accelerators-collector-config\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.357933 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.357906 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cfdce1f4-44e6-4e54-84c1-eb7f45512086-metrics-client-ca\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.359786 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.359747 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-tls\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.359913 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.359882 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cfdce1f4-44e6-4e54-84c1-eb7f45512086-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.364468 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.364445 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqwj\" (UniqueName: \"kubernetes.io/projected/cfdce1f4-44e6-4e54-84c1-eb7f45512086-kube-api-access-5fqwj\") pod \"node-exporter-svgfs\" (UID: \"cfdce1f4-44e6-4e54-84c1-eb7f45512086\") " pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.441418 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.441332 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-svgfs" Jul 30 10:49:40.892878 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.892849 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:49:40.994563 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:40.994528 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svgfs" event={"ID":"cfdce1f4-44e6-4e54-84c1-eb7f45512086","Type":"ContainerStarted","Data":"86fc4a344888384c500fafab0a0b7fd22b1d48217c3cc395e02ddebaaaf3ead0"} Jul 30 10:49:41.179888 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.179811 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 10:49:41.182692 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.182666 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.184621 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.184598 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 30 10:49:41.184951 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.184925 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 30 10:49:41.185101 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.185066 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 30 10:49:41.185187 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.185168 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7xz9m\"" Jul 30 10:49:41.185302 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.185285 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 30 10:49:41.185434 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.185412 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 30 10:49:41.185518 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.185457 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 30 10:49:41.185919 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.185750 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 30 10:49:41.185919 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.185819 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 30 10:49:41.186078 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.186054 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 30 10:49:41.199599 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.199568 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 10:49:41.367487 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367444 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.367678 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vblln\" (UniqueName: \"kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-kube-api-access-vblln\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.367678 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.367678 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-out\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.367869 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.367869 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367783 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-volume\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.367869 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.367869 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.368066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367909 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-web-config\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.368066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.368066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.368066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.367980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.368066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.368001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468433 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468405 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468556 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468440 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-volume\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468556 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468556 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468556 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-web-config\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468556 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468551 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468804 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468583 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468804 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468804 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468804 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468804 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468689 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vblln\" (UniqueName: \"kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-kube-api-access-vblln\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468804 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.468804 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.468754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-out\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.471265 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.469365 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.471265 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.469383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.471770 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.471364 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-volume\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.471770 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.471378 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.471770 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.471571 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.472509 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.471930 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-out\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.472509 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.472001 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.472509 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.472419 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-web-config\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.472996 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.472963 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.473757 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.473738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.475294 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.475270 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.475379 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.475327 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.479632 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.479601 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vblln\" (UniqueName: \"kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-kube-api-access-vblln\") pod \"alertmanager-main-0\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.496569 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.496540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:49:41.643154 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:41.643122 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 10:49:41.646212 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:41.646170 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e280f2f_9697_4b73_b27f_7a5ef1ce04af.slice/crio-3ab2cfb1c41c2e1c812817cf176b24b9a45d91067bda7fc76c97b94406358439 WatchSource:0}: Error finding container 3ab2cfb1c41c2e1c812817cf176b24b9a45d91067bda7fc76c97b94406358439: Status 404 returned error can't find the container with id 3ab2cfb1c41c2e1c812817cf176b24b9a45d91067bda7fc76c97b94406358439 Jul 30 10:49:42.000492 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:42.000457 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfdce1f4-44e6-4e54-84c1-eb7f45512086" containerID="5f3483a5bf4ff7da272d830103da1104cc7b60bea7c7b6cb0c4b7ecd13059acd" exitCode=0 Jul 30 10:49:42.000704 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:42.000551 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svgfs" event={"ID":"cfdce1f4-44e6-4e54-84c1-eb7f45512086","Type":"ContainerDied","Data":"5f3483a5bf4ff7da272d830103da1104cc7b60bea7c7b6cb0c4b7ecd13059acd"} Jul 30 10:49:42.001839 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:42.001820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerStarted","Data":"3ab2cfb1c41c2e1c812817cf176b24b9a45d91067bda7fc76c97b94406358439"} Jul 30 10:49:43.007560 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.007521 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerStarted","Data":"cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a"} Jul 30 10:49:43.010200 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.010158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svgfs" event={"ID":"cfdce1f4-44e6-4e54-84c1-eb7f45512086","Type":"ContainerStarted","Data":"1ae59ddaeebf7a6fcb452a9fccaaef4325baa7b1f87b0be5dbff07000149649f"} Jul 30 10:49:43.010323 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.010224 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-svgfs" event={"ID":"cfdce1f4-44e6-4e54-84c1-eb7f45512086","Type":"ContainerStarted","Data":"b704b6c2cc1988711154e2eb33244e13832ab77f1b4949391798a6b66f409988"} Jul 30 10:49:43.068126 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.068045 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-svgfs" podStartSLOduration=2.032004338 podStartE2EDuration="3.068025585s" totalImagesPullingTime="961.402444ms" totalInitContainerRuntime="74.618803ms" isStatefulPod=true podCreationTimestamp="2026-07-30 10:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:43.067767044 +0000 UTC m=+91.040628103" watchObservedRunningTime="2026-07-30 10:49:43.068025585 +0000 UTC m=+91.040886621" Jul 30 10:49:43.206486 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.206402 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq"] Jul 30 10:49:43.208965 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.208947 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.211029 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.211003 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nrzm2\"" Jul 30 10:49:43.211188 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.211160 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 30 10:49:43.211550 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.211496 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 30 10:49:43.211550 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.211535 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 30 10:49:43.211704 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.211566 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7ub3of5eetojg\"" Jul 30 10:49:43.211704 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.211644 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 30 10:49:43.212082 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.212067 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 30 10:49:43.226720 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.226688 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq"] Jul 30 10:49:43.386818 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.386774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.386984 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.386841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.386984 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.386867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/581526e9-9662-47f1-97d5-7996788a978e-metrics-client-ca\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.387141 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.386977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-grpc-tls\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.387141 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.387023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-tls\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.387141 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.387053 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.387289 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.387164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x748t\" (UniqueName: \"kubernetes.io/projected/581526e9-9662-47f1-97d5-7996788a978e-kube-api-access-x748t\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.387289 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.387205 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.488011 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.487911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x748t\" (UniqueName: \"kubernetes.io/projected/581526e9-9662-47f1-97d5-7996788a978e-kube-api-access-x748t\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.488011 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.487961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.488011 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.487994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.488330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.488020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.488330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.488040 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/581526e9-9662-47f1-97d5-7996788a978e-metrics-client-ca\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.488330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.488112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-grpc-tls\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.488330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.488151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-tls\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.488330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.488193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.488902 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.488873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/581526e9-9662-47f1-97d5-7996788a978e-metrics-client-ca\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.490915 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.490888 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.491058 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.490897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.491176 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.491158 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.491258 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.491243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.491327 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.491299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-grpc-tls\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.491422 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.491402 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/581526e9-9662-47f1-97d5-7996788a978e-secret-thanos-querier-tls\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.496635 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.496617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x748t\" (UniqueName: \"kubernetes.io/projected/581526e9-9662-47f1-97d5-7996788a978e-kube-api-access-x748t\") pod \"thanos-querier-7c7dc8f8bb-cxqtq\" (UID: \"581526e9-9662-47f1-97d5-7996788a978e\") " pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.517847 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.517807 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:43.657806 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.657759 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq"] Jul 30 10:49:43.660976 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:49:43.660948 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581526e9_9662_47f1_97d5_7996788a978e.slice/crio-bea936bf02222c6246918e83bb40768c3c95d7583285046229213edf60345d0b WatchSource:0}: Error finding container bea936bf02222c6246918e83bb40768c3c95d7583285046229213edf60345d0b: Status 404 returned error can't find the container with id bea936bf02222c6246918e83bb40768c3c95d7583285046229213edf60345d0b Jul 30 10:49:43.902678 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:43.902643 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bcb89d8c9-shq24"] Jul 30 10:49:44.015215 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:44.015178 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" event={"ID":"581526e9-9662-47f1-97d5-7996788a978e","Type":"ContainerStarted","Data":"bea936bf02222c6246918e83bb40768c3c95d7583285046229213edf60345d0b"} Jul 30 10:49:44.016789 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:44.016762 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerID="cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a" exitCode=0 Jul 30 10:49:44.016919 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:44.016852 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerDied","Data":"cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a"} Jul 30 10:49:46.032471 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:46.032423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerStarted","Data":"2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f"} Jul 30 10:49:46.032900 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:46.032542 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerStarted","Data":"8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd"} Jul 30 10:49:46.032900 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:46.032587 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerStarted","Data":"99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb"} Jul 30 10:49:46.032900 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:46.032629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerStarted","Data":"174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c"} Jul 30 10:49:46.032900 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:46.032666 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerStarted","Data":"73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b"} Jul 30 10:49:48.045388 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:48.045357 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerStarted","Data":"d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d"} Jul 30 10:49:48.047446 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:48.047420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" event={"ID":"581526e9-9662-47f1-97d5-7996788a978e","Type":"ContainerStarted","Data":"8fb82ad1ba821854d9d476d953e4598734ff8df244372d6b79256da7c6265c68"} Jul 30 10:49:48.047588 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:48.047493 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" event={"ID":"581526e9-9662-47f1-97d5-7996788a978e","Type":"ContainerStarted","Data":"23d353703f8b6c53dd3de6d5553e18d8989415a375d048b1171a8d0f51d3d45c"} Jul 30 10:49:48.074360 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:48.074306 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.3849807800000002 podStartE2EDuration="7.07428433s" totalImagesPullingTime="4.664214603s" totalInitContainerRuntime="25.088947ms" isStatefulPod=false podCreationTimestamp="2026-07-30 10:49:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:48.073802238 +0000 UTC m=+96.046663315" watchObservedRunningTime="2026-07-30 10:49:48.07428433 +0000 UTC m=+96.047145365" Jul 30 10:49:49.055823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:49.055774 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" event={"ID":"581526e9-9662-47f1-97d5-7996788a978e","Type":"ContainerStarted","Data":"57efec3ad7e0fab0a8a0974c21b7232a8921e228eabed181892054843004e822"} Jul 30 10:49:49.056374 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:49.055844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" event={"ID":"581526e9-9662-47f1-97d5-7996788a978e","Type":"ContainerStarted","Data":"a54fdc33bfca800ecbbb9fddbd3e0e174b082796ec49672a0842f02677d14d4e"} Jul 30 10:49:49.056374 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:49.055870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" event={"ID":"581526e9-9662-47f1-97d5-7996788a978e","Type":"ContainerStarted","Data":"5d45aba6000531e3c805b12ae78c1b51206d5ab5059f93849bb1d293003105e2"} Jul 30 10:49:49.056374 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:49.055893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" event={"ID":"581526e9-9662-47f1-97d5-7996788a978e","Type":"ContainerStarted","Data":"ce0475950b24cb120ca196e07affdb5dd00886cca5bab9d8571b25f49bb69142"} Jul 30 10:49:49.078591 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:49.078525 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" podStartSLOduration=2.005647755 podStartE2EDuration="6.078504341s" totalImagesPullingTime="4.072856586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:49:49.07713237 +0000 UTC m=+97.049993419" watchObservedRunningTime="2026-07-30 10:49:49.078504341 +0000 UTC m=+97.051365377" Jul 30 10:49:50.061130 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:50.061082 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:49:50.893325 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:50.893292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cdb5g" Jul 30 10:49:56.071920 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:49:56.071892 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c7dc8f8bb-cxqtq" Jul 30 10:50:08.921976 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:08.921935 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" podUID="98410b9f-a736-487f-8c82-b3df27f5113b" containerName="registry" containerID="cri-o://748e5c41fd96a2b8cc9b6f3703d65cd2177bce19329f6c4da158e1bbb4ba86cc" gracePeriod=30 Jul 30 10:50:09.136637 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.136604 2575 generic.go:356] "Generic (PLEG): container finished" podID="98410b9f-a736-487f-8c82-b3df27f5113b" containerID="748e5c41fd96a2b8cc9b6f3703d65cd2177bce19329f6c4da158e1bbb4ba86cc" exitCode=0 Jul 30 10:50:09.136796 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.136675 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" event={"ID":"98410b9f-a736-487f-8c82-b3df27f5113b","Type":"ContainerDied","Data":"748e5c41fd96a2b8cc9b6f3703d65cd2177bce19329f6c4da158e1bbb4ba86cc"} Jul 30 10:50:09.166857 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.166836 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:50:09.206608 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.206532 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98410b9f-a736-487f-8c82-b3df27f5113b-ca-trust-extracted\") pod \"98410b9f-a736-487f-8c82-b3df27f5113b\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " Jul 30 10:50:09.206608 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.206569 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-installation-pull-secrets\") pod \"98410b9f-a736-487f-8c82-b3df27f5113b\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " Jul 30 10:50:09.206608 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.206594 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-registry-certificates\") pod \"98410b9f-a736-487f-8c82-b3df27f5113b\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " Jul 30 10:50:09.206877 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.206621 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czhjf\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-kube-api-access-czhjf\") pod \"98410b9f-a736-487f-8c82-b3df27f5113b\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " Jul 30 10:50:09.206877 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.206675 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-trusted-ca\") pod \"98410b9f-a736-487f-8c82-b3df27f5113b\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " Jul 30 10:50:09.206978 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.206906 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") pod \"98410b9f-a736-487f-8c82-b3df27f5113b\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " Jul 30 10:50:09.206978 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.206955 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-image-registry-private-configuration\") pod \"98410b9f-a736-487f-8c82-b3df27f5113b\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " Jul 30 10:50:09.207079 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.206993 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-bound-sa-token\") pod \"98410b9f-a736-487f-8c82-b3df27f5113b\" (UID: \"98410b9f-a736-487f-8c82-b3df27f5113b\") " Jul 30 10:50:09.207168 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.207133 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "98410b9f-a736-487f-8c82-b3df27f5113b" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 10:50:09.207324 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.207305 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-trusted-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:50:09.207324 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.207129 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "98410b9f-a736-487f-8c82-b3df27f5113b" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 10:50:09.209559 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.209476 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-kube-api-access-czhjf" (OuterVolumeSpecName: "kube-api-access-czhjf") pod "98410b9f-a736-487f-8c82-b3df27f5113b" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b"). InnerVolumeSpecName "kube-api-access-czhjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:50:09.209559 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.209509 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "98410b9f-a736-487f-8c82-b3df27f5113b" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:50:09.209741 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.209570 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "98410b9f-a736-487f-8c82-b3df27f5113b" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:50:09.209807 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.209782 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "98410b9f-a736-487f-8c82-b3df27f5113b" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:50:09.209885 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.209864 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "98410b9f-a736-487f-8c82-b3df27f5113b" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:50:09.215486 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.215457 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98410b9f-a736-487f-8c82-b3df27f5113b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "98410b9f-a736-487f-8c82-b3df27f5113b" (UID: "98410b9f-a736-487f-8c82-b3df27f5113b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:50:09.307833 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.307781 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-registry-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:50:09.307833 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.307829 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-image-registry-private-configuration\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:50:09.307833 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.307838 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-bound-sa-token\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:50:09.307833 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.307847 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/98410b9f-a736-487f-8c82-b3df27f5113b-ca-trust-extracted\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:50:09.307833 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.307856 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/98410b9f-a736-487f-8c82-b3df27f5113b-installation-pull-secrets\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:50:09.308176 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.307865 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/98410b9f-a736-487f-8c82-b3df27f5113b-registry-certificates\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:50:09.308176 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:09.307873 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czhjf\" (UniqueName: \"kubernetes.io/projected/98410b9f-a736-487f-8c82-b3df27f5113b-kube-api-access-czhjf\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:50:10.141661 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:10.141632 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" Jul 30 10:50:10.142177 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:10.141670 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bcb89d8c9-shq24" event={"ID":"98410b9f-a736-487f-8c82-b3df27f5113b","Type":"ContainerDied","Data":"2314539a2d81e1694e23f689d91c705fe6b0ddc942bcda37d8f1a2d0949af2a8"} Jul 30 10:50:10.142177 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:10.141733 2575 scope.go:117] "RemoveContainer" containerID="748e5c41fd96a2b8cc9b6f3703d65cd2177bce19329f6c4da158e1bbb4ba86cc" Jul 30 10:50:10.161345 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:10.161317 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bcb89d8c9-shq24"] Jul 30 10:50:10.164038 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:10.164018 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bcb89d8c9-shq24"] Jul 30 10:50:10.525591 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:10.525501 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98410b9f-a736-487f-8c82-b3df27f5113b" path="/var/lib/kubelet/pods/98410b9f-a736-487f-8c82-b3df27f5113b/volumes" Jul 30 10:50:17.170750 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:17.170716 2575 generic.go:356] "Generic (PLEG): container finished" podID="829982dc-21f7-4ee7-accb-0182ccd29add" containerID="291d6b17191f5b5f101221316db2e1141bb2765f09ad1875caca5bbd94393349" exitCode=0 Jul 30 10:50:17.171229 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:17.170804 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-h8zf7" event={"ID":"829982dc-21f7-4ee7-accb-0182ccd29add","Type":"ContainerDied","Data":"291d6b17191f5b5f101221316db2e1141bb2765f09ad1875caca5bbd94393349"} Jul 30 10:50:17.171229 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:17.171195 2575 scope.go:117] "RemoveContainer" containerID="291d6b17191f5b5f101221316db2e1141bb2765f09ad1875caca5bbd94393349" Jul 30 10:50:17.172234 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:17.172208 2575 generic.go:356] "Generic (PLEG): container finished" podID="d78e2d15-951a-4df2-afce-ce74eb971837" containerID="0700d77284ceb33de25e9709e3e32f4611387be791e1ee7f5653495e8f26baf8" exitCode=0 Jul 30 10:50:17.172333 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:17.172287 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" event={"ID":"d78e2d15-951a-4df2-afce-ce74eb971837","Type":"ContainerDied","Data":"0700d77284ceb33de25e9709e3e32f4611387be791e1ee7f5653495e8f26baf8"} Jul 30 10:50:17.172716 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:17.172597 2575 scope.go:117] "RemoveContainer" containerID="0700d77284ceb33de25e9709e3e32f4611387be791e1ee7f5653495e8f26baf8" Jul 30 10:50:17.179250 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:17.174265 2575 generic.go:356] "Generic (PLEG): container finished" podID="7d5c371a-680e-4c9b-aac7-aa5c13018096" containerID="2b4cc9afc5c128e734eb4a933666ed4a4f5708aef666ff450082f8a59b930d7b" exitCode=0 Jul 30 10:50:17.179250 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:17.174311 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" event={"ID":"7d5c371a-680e-4c9b-aac7-aa5c13018096","Type":"ContainerDied","Data":"2b4cc9afc5c128e734eb4a933666ed4a4f5708aef666ff450082f8a59b930d7b"} Jul 30 10:50:17.179250 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:17.174699 2575 scope.go:117] "RemoveContainer" containerID="2b4cc9afc5c128e734eb4a933666ed4a4f5708aef666ff450082f8a59b930d7b" Jul 30 10:50:18.180359 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:18.180322 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-f98ms" event={"ID":"d78e2d15-951a-4df2-afce-ce74eb971837","Type":"ContainerStarted","Data":"f33566e6470236e314bd88ba4677c2159395f4ce9fc2bd3cdf027d5b8508d74a"} Jul 30 10:50:18.182219 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:18.182196 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx" event={"ID":"7d5c371a-680e-4c9b-aac7-aa5c13018096","Type":"ContainerStarted","Data":"9e0b790bff4302b2773139186a88e3633d8de7ad2aec84f33e90b8e26a37bda6"} Jul 30 10:50:18.184071 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:50:18.184048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-h8zf7" event={"ID":"829982dc-21f7-4ee7-accb-0182ccd29add","Type":"ContainerStarted","Data":"31942e7acb7cd382cb7d76b8f8e1cec0c4c068cfac84e4c70dd09037f360e395"} Jul 30 10:51:00.460680 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:00.460626 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 10:51:00.461286 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:00.461229 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="alertmanager" containerID="cri-o://73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b" gracePeriod=120 Jul 30 10:51:00.463116 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:00.461501 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy" containerID="cri-o://8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd" gracePeriod=120 Jul 30 10:51:00.463116 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:00.461671 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy-web" containerID="cri-o://99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb" gracePeriod=120 Jul 30 10:51:00.463116 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:00.461707 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy-metric" containerID="cri-o://2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f" gracePeriod=120 Jul 30 10:51:00.463116 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:00.461781 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="config-reloader" containerID="cri-o://174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c" gracePeriod=120 Jul 30 10:51:00.463116 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:00.461679 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="prom-label-proxy" containerID="cri-o://d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d" gracePeriod=120 Jul 30 10:51:01.364037 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.364005 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerID="d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d" exitCode=0 Jul 30 10:51:01.364037 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.364031 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerID="8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd" exitCode=0 Jul 30 10:51:01.364037 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.364039 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerID="174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c" exitCode=0 Jul 30 10:51:01.364037 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.364047 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerID="73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b" exitCode=0 Jul 30 10:51:01.364348 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.364080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerDied","Data":"d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d"} Jul 30 10:51:01.364348 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.364165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerDied","Data":"8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd"} Jul 30 10:51:01.364348 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.364192 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerDied","Data":"174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c"} Jul 30 10:51:01.364348 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.364217 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerDied","Data":"73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b"} Jul 30 10:51:01.703601 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.703574 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:01.748629 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748596 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-volume\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.748810 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748652 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-trusted-ca-bundle\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.748810 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748717 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.748810 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748744 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-web-config\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.748810 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748807 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-out\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.749019 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748837 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-web\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.749019 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748866 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-main-tls\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.749019 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748893 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-tls-assets\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.749019 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748915 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-metrics-client-ca\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.749019 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748944 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.749019 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748971 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-cluster-tls-config\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.749019 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.748999 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vblln\" (UniqueName: \"kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-kube-api-access-vblln\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.749386 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.749031 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-main-db\") pod \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\" (UID: \"2e280f2f-9697-4b73-b27f-7a5ef1ce04af\") " Jul 30 10:51:01.749755 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.749578 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:51:01.749755 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.749667 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 10:51:01.749919 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.749830 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 10:51:01.752072 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.751998 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:51:01.754223 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.754191 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-kube-api-access-vblln" (OuterVolumeSpecName: "kube-api-access-vblln") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "kube-api-access-vblln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:51:01.754315 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.754264 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:51:01.754384 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.754350 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:51:01.754522 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.754486 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:51:01.754623 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.754597 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-volume" (OuterVolumeSpecName: "config-volume") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:51:01.754912 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.754879 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:51:01.755022 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.754914 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-out" (OuterVolumeSpecName: "config-out") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:51:01.759502 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.759472 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:51:01.767420 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.767391 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-web-config" (OuterVolumeSpecName: "web-config") pod "2e280f2f-9697-4b73-b27f-7a5ef1ce04af" (UID: "2e280f2f-9697-4b73-b27f-7a5ef1ce04af"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:51:01.850050 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850011 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850050 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850042 2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-web-config\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850050 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850056 2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-out\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850068 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850083 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-main-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850120 2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-tls-assets\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850133 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-metrics-client-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850144 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850157 2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-cluster-tls-config\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850168 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vblln\" (UniqueName: \"kubernetes.io/projected/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-kube-api-access-vblln\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850179 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-main-db\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850190 2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:01.850328 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:01.850202 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e280f2f-9697-4b73-b27f-7a5ef1ce04af-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:51:02.371080 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.371044 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerID="2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f" exitCode=0 Jul 30 10:51:02.371080 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.371067 2575 generic.go:356] "Generic (PLEG): container finished" podID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerID="99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb" exitCode=0 Jul 30 10:51:02.371318 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.371117 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerDied","Data":"2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f"} Jul 30 10:51:02.371318 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.371132 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.371318 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.371232 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerDied","Data":"99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb"} Jul 30 10:51:02.371318 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.371264 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e280f2f-9697-4b73-b27f-7a5ef1ce04af","Type":"ContainerDied","Data":"3ab2cfb1c41c2e1c812817cf176b24b9a45d91067bda7fc76c97b94406358439"} Jul 30 10:51:02.371318 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.371300 2575 scope.go:117] "RemoveContainer" containerID="d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d" Jul 30 10:51:02.380347 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.380313 2575 scope.go:117] "RemoveContainer" containerID="2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f" Jul 30 10:51:02.388063 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.388046 2575 scope.go:117] "RemoveContainer" containerID="8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd" Jul 30 10:51:02.395331 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.395309 2575 scope.go:117] "RemoveContainer" containerID="99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb" Jul 30 10:51:02.396536 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.396486 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 10:51:02.402117 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.402077 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 10:51:02.403844 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.403829 2575 scope.go:117] "RemoveContainer" containerID="174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c" Jul 30 10:51:02.410819 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.410793 2575 scope.go:117] "RemoveContainer" containerID="73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b" Jul 30 10:51:02.418138 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.418119 2575 scope.go:117] "RemoveContainer" containerID="cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a" Jul 30 10:51:02.425470 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.425449 2575 scope.go:117] "RemoveContainer" containerID="d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d" Jul 30 10:51:02.425763 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:51:02.425742 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d\": container with ID starting with d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d not found: ID does not exist" containerID="d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d" Jul 30 10:51:02.425817 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.425772 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d"} err="failed to get container status \"d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d\": rpc error: code = NotFound desc = could not find container \"d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d\": container with ID starting with d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d not found: ID does not exist" Jul 30 10:51:02.425858 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.425816 2575 scope.go:117] "RemoveContainer" containerID="2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f" Jul 30 10:51:02.426071 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:51:02.426054 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f\": container with ID starting with 2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f not found: ID does not exist" containerID="2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f" Jul 30 10:51:02.426153 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.426080 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f"} err="failed to get container status \"2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f\": rpc error: code = NotFound desc = could not find container \"2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f\": container with ID starting with 2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f not found: ID does not exist" Jul 30 10:51:02.426153 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.426122 2575 scope.go:117] "RemoveContainer" containerID="8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd" Jul 30 10:51:02.426365 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:51:02.426348 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd\": container with ID starting with 8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd not found: ID does not exist" containerID="8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd" Jul 30 10:51:02.426404 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.426369 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd"} err="failed to get container status \"8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd\": rpc error: code = NotFound desc = could not find container \"8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd\": container with ID starting with 8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd not found: ID does not exist" Jul 30 10:51:02.426404 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.426386 2575 scope.go:117] "RemoveContainer" containerID="99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb" Jul 30 10:51:02.426603 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:51:02.426587 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb\": container with ID starting with 99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb not found: ID does not exist" containerID="99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb" Jul 30 10:51:02.426644 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.426608 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb"} err="failed to get container status \"99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb\": rpc error: code = NotFound desc = could not find container \"99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb\": container with ID starting with 99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb not found: ID does not exist" Jul 30 10:51:02.426644 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.426624 2575 scope.go:117] "RemoveContainer" containerID="174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c" Jul 30 10:51:02.426855 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:51:02.426839 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c\": container with ID starting with 174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c not found: ID does not exist" containerID="174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c" Jul 30 10:51:02.426897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.426860 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c"} err="failed to get container status \"174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c\": rpc error: code = NotFound desc = could not find container \"174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c\": container with ID starting with 174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c not found: ID does not exist" Jul 30 10:51:02.426897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.426872 2575 scope.go:117] "RemoveContainer" containerID="73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b" Jul 30 10:51:02.427075 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:51:02.427056 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b\": container with ID starting with 73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b not found: ID does not exist" containerID="73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b" Jul 30 10:51:02.427172 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.427082 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b"} err="failed to get container status \"73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b\": rpc error: code = NotFound desc = could not find container \"73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b\": container with ID starting with 73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b not found: ID does not exist" Jul 30 10:51:02.427172 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.427115 2575 scope.go:117] "RemoveContainer" containerID="cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a" Jul 30 10:51:02.427337 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:51:02.427322 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a\": container with ID starting with cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a not found: ID does not exist" containerID="cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a" Jul 30 10:51:02.427377 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.427340 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a"} err="failed to get container status \"cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a\": rpc error: code = NotFound desc = could not find container \"cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a\": container with ID starting with cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a not found: ID does not exist" Jul 30 10:51:02.427377 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.427351 2575 scope.go:117] "RemoveContainer" containerID="d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d" Jul 30 10:51:02.427586 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.427564 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d"} err="failed to get container status \"d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d\": rpc error: code = NotFound desc = could not find container \"d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d\": container with ID starting with d19073f06b267a2d7ac520565f4aa13fc1fcb0e89b69544e0d52be6803aca18d not found: ID does not exist" Jul 30 10:51:02.427586 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.427586 2575 scope.go:117] "RemoveContainer" containerID="2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f" Jul 30 10:51:02.427818 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.427801 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f"} err="failed to get container status \"2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f\": rpc error: code = NotFound desc = could not find container \"2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f\": container with ID starting with 2ce287ed46c3576f665042933e5c16d87815553309d5b30bdddad887b7a0153f not found: ID does not exist" Jul 30 10:51:02.427863 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.427818 2575 scope.go:117] "RemoveContainer" containerID="8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd" Jul 30 10:51:02.428039 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.428021 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd"} err="failed to get container status \"8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd\": rpc error: code = NotFound desc = could not find container \"8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd\": container with ID starting with 8e610062e4034f12f2d216760ff4ebf963504ca53b9e735a260eed2f86e20bfd not found: ID does not exist" Jul 30 10:51:02.428039 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.428038 2575 scope.go:117] "RemoveContainer" containerID="99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb" Jul 30 10:51:02.428246 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.428230 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb"} err="failed to get container status \"99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb\": rpc error: code = NotFound desc = could not find container \"99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb\": container with ID starting with 99fe3df7107f528bb1ebbb7bb04b978ae7a455fbba7c1e558087eeb6a23709eb not found: ID does not exist" Jul 30 10:51:02.428293 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.428246 2575 scope.go:117] "RemoveContainer" containerID="174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c" Jul 30 10:51:02.428442 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.428426 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c"} err="failed to get container status \"174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c\": rpc error: code = NotFound desc = could not find container \"174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c\": container with ID starting with 174e8cfea9cfe1825e57b3372d752a1ff04b55406b7f8b53c6592a3791bf6a7c not found: ID does not exist" Jul 30 10:51:02.428484 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.428442 2575 scope.go:117] "RemoveContainer" containerID="73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b" Jul 30 10:51:02.428618 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.428602 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b"} err="failed to get container status \"73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b\": rpc error: code = NotFound desc = could not find container \"73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b\": container with ID starting with 73133c0a9479c6bdbacf467466115448e030a5534a04929be29a167e7699137b not found: ID does not exist" Jul 30 10:51:02.428657 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.428617 2575 scope.go:117] "RemoveContainer" containerID="cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a" Jul 30 10:51:02.428811 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.428796 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a"} err="failed to get container status \"cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a\": rpc error: code = NotFound desc = could not find container \"cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a\": container with ID starting with cad5e81db7f01516ec90502f94d68bcc3437a7a2536548f1ad061a1ea462e86a not found: ID does not exist" Jul 30 10:51:02.433256 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433234 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 10:51:02.433580 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433564 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy-web" Jul 30 10:51:02.433653 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433582 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy-web" Jul 30 10:51:02.433653 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433596 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy" Jul 30 10:51:02.433653 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433604 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy" Jul 30 10:51:02.433653 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433628 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="config-reloader" Jul 30 10:51:02.433653 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433638 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="config-reloader" Jul 30 10:51:02.433653 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433647 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy-metric" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433656 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy-metric" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433670 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="alertmanager" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433679 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="alertmanager" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433693 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98410b9f-a736-487f-8c82-b3df27f5113b" containerName="registry" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433701 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="98410b9f-a736-487f-8c82-b3df27f5113b" containerName="registry" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433711 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="init-config-reloader" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433719 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="init-config-reloader" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433736 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="prom-label-proxy" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433744 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="prom-label-proxy" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433824 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy-metric" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433834 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="prom-label-proxy" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433845 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="alertmanager" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433853 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="98410b9f-a736-487f-8c82-b3df27f5113b" containerName="registry" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433862 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="config-reloader" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433870 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy" Jul 30 10:51:02.433939 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.433878 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" containerName="kube-rbac-proxy-web" Jul 30 10:51:02.439175 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.439156 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.441168 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.441148 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 30 10:51:02.441278 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.441259 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 30 10:51:02.441547 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.441260 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 30 10:51:02.441547 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.441439 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 30 10:51:02.441547 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.441499 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 30 10:51:02.441743 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.441724 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7xz9m\"" Jul 30 10:51:02.441801 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.441748 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 30 10:51:02.442187 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.442170 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 30 10:51:02.442317 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.442301 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 30 10:51:02.447478 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.447460 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 30 10:51:02.452576 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.452557 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 10:51:02.526043 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.526010 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e280f2f-9697-4b73-b27f-7a5ef1ce04af" path="/var/lib/kubelet/pods/2e280f2f-9697-4b73-b27f-7a5ef1ce04af/volumes" Jul 30 10:51:02.555715 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.555670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.555715 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.555716 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhww7\" (UniqueName: \"kubernetes.io/projected/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-kube-api-access-mhww7\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.555973 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.555737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-config-volume\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.555973 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.555791 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-web-config\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.555973 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.555829 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.555973 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.555874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.555973 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.555907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.555973 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.555934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.555973 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.555961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.556284 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.556010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-config-out\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.556284 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.556036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.556284 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.556117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.556284 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.556143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657160 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657160 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657160 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657161 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657417 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657417 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657238 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhww7\" (UniqueName: \"kubernetes.io/projected/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-kube-api-access-mhww7\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657417 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-config-volume\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657417 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-web-config\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657417 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657417 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657417 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657417 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657697 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.657697 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.657463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-config-out\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.658461 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.658431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.660728 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.660634 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.660728 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.660669 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-config-out\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.660728 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.660635 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.661083 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.661058 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-config-volume\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.661198 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.661177 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.661907 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.661879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.662181 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.662159 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.664135 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.663564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.664135 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.663731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.664135 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.663956 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-web-config\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.666485 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.665567 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.666935 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.666907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhww7\" (UniqueName: \"kubernetes.io/projected/0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8-kube-api-access-mhww7\") pod \"alertmanager-main-0\" (UID: \"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.749712 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.749665 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 10:51:02.893453 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:02.893421 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 10:51:02.896947 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:51:02.896922 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a65dba3_3f5a_4a3e_bdbb_a43724fb0cf8.slice/crio-e85ced8bb974de2eaac17dc5dd4c2202c432a3c244dfab8ff4ed90b652d32c26 WatchSource:0}: Error finding container e85ced8bb974de2eaac17dc5dd4c2202c432a3c244dfab8ff4ed90b652d32c26: Status 404 returned error can't find the container with id e85ced8bb974de2eaac17dc5dd4c2202c432a3c244dfab8ff4ed90b652d32c26 Jul 30 10:51:03.377202 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:03.377164 2575 generic.go:356] "Generic (PLEG): container finished" podID="0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8" containerID="5d9818586fcb0fd02f4068b4895b662c237f7d04cdc9b01bbb3ab88e3d19194d" exitCode=0 Jul 30 10:51:03.377364 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:03.377256 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8","Type":"ContainerDied","Data":"5d9818586fcb0fd02f4068b4895b662c237f7d04cdc9b01bbb3ab88e3d19194d"} Jul 30 10:51:03.377364 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:03.377318 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8","Type":"ContainerStarted","Data":"e85ced8bb974de2eaac17dc5dd4c2202c432a3c244dfab8ff4ed90b652d32c26"} Jul 30 10:51:04.386013 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.385973 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8","Type":"ContainerStarted","Data":"45a20f1c9a061d725dbb30c5248ab07825bec928a06fec190942111e082646b9"} Jul 30 10:51:04.386536 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.386065 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8","Type":"ContainerStarted","Data":"3241520cbc8aa36003bdfd6b7c9fee010304f6b06262c817283c08929b1c2d29"} Jul 30 10:51:04.386536 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.386126 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8","Type":"ContainerStarted","Data":"f56ea19fcd7e12ddcd6e7a3ca7b468849b9c0da8806de73ae31545594a001b19"} Jul 30 10:51:04.386536 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.386160 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8","Type":"ContainerStarted","Data":"98ee321c7a5ab3ddcad90980d2ea09dbc88b44fd0818e43f6ec34eed7d59bb3a"} Jul 30 10:51:04.386536 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.386184 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8","Type":"ContainerStarted","Data":"9259385f652daa3b744470e57e11ef51b71167287e7a9b0f1072d1e9dc593aba"} Jul 30 10:51:04.386536 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.386221 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8","Type":"ContainerStarted","Data":"c06d3184051d710cdb2c4e673f1c4169c49c514f5a279a1403fffba404c1e4b0"} Jul 30 10:51:04.421813 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.421744 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.406153129 podStartE2EDuration="2.42172191s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.568781ms" isStatefulPod=false podCreationTimestamp="2026-07-30 10:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:51:04.418645071 +0000 UTC m=+172.391506150" watchObservedRunningTime="2026-07-30 10:51:04.42172191 +0000 UTC m=+172.394582948" Jul 30 10:51:04.532792 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.532753 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn"] Jul 30 10:51:04.538390 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.538365 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.541072 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.541040 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bdmpf\"" Jul 30 10:51:04.541483 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.540632 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 30 10:51:04.541994 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.541968 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 30 10:51:04.544325 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.542255 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 30 10:51:04.544325 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.542519 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 30 10:51:04.544325 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.542771 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 30 10:51:04.550379 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.550354 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 30 10:51:04.553049 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.553020 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn"] Jul 30 10:51:04.674292 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.674182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a061586f-9efd-456e-a0ad-675b136287ec-serving-certs-ca-bundle\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.674292 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.674252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a061586f-9efd-456e-a0ad-675b136287ec-metrics-client-ca\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.674509 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.674299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.674509 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.674322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjnh\" (UniqueName: \"kubernetes.io/projected/a061586f-9efd-456e-a0ad-675b136287ec-kube-api-access-hfjnh\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.674509 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.674359 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-secret-telemeter-client\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.674509 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.674391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-federate-client-tls\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.674509 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.674435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-telemeter-client-tls\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.674509 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.674464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a061586f-9efd-456e-a0ad-675b136287ec-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.775315 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.775282 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a061586f-9efd-456e-a0ad-675b136287ec-metrics-client-ca\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.775528 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.775335 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.775528 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.775357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjnh\" (UniqueName: \"kubernetes.io/projected/a061586f-9efd-456e-a0ad-675b136287ec-kube-api-access-hfjnh\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.775528 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.775385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-secret-telemeter-client\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.775528 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.775407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-federate-client-tls\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.775709 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.775594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-telemeter-client-tls\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.775709 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.775643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a061586f-9efd-456e-a0ad-675b136287ec-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.775709 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.775681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a061586f-9efd-456e-a0ad-675b136287ec-serving-certs-ca-bundle\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.776359 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.776269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a061586f-9efd-456e-a0ad-675b136287ec-metrics-client-ca\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.776482 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.776454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a061586f-9efd-456e-a0ad-675b136287ec-serving-certs-ca-bundle\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.776670 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.776646 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a061586f-9efd-456e-a0ad-675b136287ec-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.778049 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.778027 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-federate-client-tls\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.778231 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.778194 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.778354 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.778240 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-telemeter-client-tls\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.778426 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.778383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a061586f-9efd-456e-a0ad-675b136287ec-secret-telemeter-client\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.785582 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.784832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjnh\" (UniqueName: \"kubernetes.io/projected/a061586f-9efd-456e-a0ad-675b136287ec-kube-api-access-hfjnh\") pod \"telemeter-client-5c4b78bbc6-jbxdn\" (UID: \"a061586f-9efd-456e-a0ad-675b136287ec\") " pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:04.853907 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:04.853867 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" Jul 30 10:51:05.017302 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:05.017261 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn"] Jul 30 10:51:05.021026 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:51:05.020993 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda061586f_9efd_456e_a0ad_675b136287ec.slice/crio-6f40a845e59aa991b4b185773b5008e640be790c22dd54e5dd5f20444b4e907f WatchSource:0}: Error finding container 6f40a845e59aa991b4b185773b5008e640be790c22dd54e5dd5f20444b4e907f: Status 404 returned error can't find the container with id 6f40a845e59aa991b4b185773b5008e640be790c22dd54e5dd5f20444b4e907f Jul 30 10:51:05.391725 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:05.391690 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" event={"ID":"a061586f-9efd-456e-a0ad-675b136287ec","Type":"ContainerStarted","Data":"6f40a845e59aa991b4b185773b5008e640be790c22dd54e5dd5f20444b4e907f"} Jul 30 10:51:07.403152 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:07.403114 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" event={"ID":"a061586f-9efd-456e-a0ad-675b136287ec","Type":"ContainerStarted","Data":"0a781d5ff25864be104a85e69bd1133b3e16922954e1c8f8963f3789f2014847"} Jul 30 10:51:07.403563 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:07.403180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" event={"ID":"a061586f-9efd-456e-a0ad-675b136287ec","Type":"ContainerStarted","Data":"dab2e12ffdb9d6f984cbc447c0e090c7d55b8f31fe69d1f8825a00194e92ad0e"} Jul 30 10:51:07.403563 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:07.403206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" event={"ID":"a061586f-9efd-456e-a0ad-675b136287ec","Type":"ContainerStarted","Data":"843dd9b8b7d84fd84defb43dc8385e75ca087e67f6f6f6bd47c24a77ab329b28"} Jul 30 10:51:07.427774 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:51:07.427719 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5c4b78bbc6-jbxdn" podStartSLOduration=1.5468394540000001 podStartE2EDuration="3.427701169s" totalImagesPullingTime="1.880861715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:51:07.425581635 +0000 UTC m=+175.398442670" watchObservedRunningTime="2026-07-30 10:51:07.427701169 +0000 UTC m=+175.400562205" Jul 30 10:53:12.412690 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:12.412653 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 10:53:12.418802 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:12.418772 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 10:53:12.422565 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:12.422546 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 10:53:23.376681 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.376644 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t"] Jul 30 10:53:23.379850 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.379835 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.382129 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.382077 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 10:53:23.382269 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.382141 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 10:53:23.382477 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.382459 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ldxh6\"" Jul 30 10:53:23.395383 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.395358 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t"] Jul 30 10:53:23.459405 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.459363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8cl\" (UniqueName: \"kubernetes.io/projected/bd12f693-9299-40f2-aa8a-ee3afc279807-kube-api-access-dk8cl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.459582 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.459416 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.459582 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.459488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.560716 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.560680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8cl\" (UniqueName: \"kubernetes.io/projected/bd12f693-9299-40f2-aa8a-ee3afc279807-kube-api-access-dk8cl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.560864 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.560731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.560864 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.560801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.561268 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.561247 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.561355 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.561285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.570397 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.570372 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8cl\" (UniqueName: \"kubernetes.io/projected/bd12f693-9299-40f2-aa8a-ee3afc279807-kube-api-access-dk8cl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.690743 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.690659 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:23.822953 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.822911 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t"] Jul 30 10:53:23.826022 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:53:23.825994 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd12f693_9299_40f2_aa8a_ee3afc279807.slice/crio-a3a1a9cbac0f03accb7f3a5fada0eefb30c55d751ea40c794b2ce2b4a14f6442 WatchSource:0}: Error finding container a3a1a9cbac0f03accb7f3a5fada0eefb30c55d751ea40c794b2ce2b4a14f6442: Status 404 returned error can't find the container with id a3a1a9cbac0f03accb7f3a5fada0eefb30c55d751ea40c794b2ce2b4a14f6442 Jul 30 10:53:23.828438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.828415 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 10:53:23.952292 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:23.952204 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" event={"ID":"bd12f693-9299-40f2-aa8a-ee3afc279807","Type":"ContainerStarted","Data":"a3a1a9cbac0f03accb7f3a5fada0eefb30c55d751ea40c794b2ce2b4a14f6442"} Jul 30 10:53:30.983346 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:30.983306 2575 generic.go:356] "Generic (PLEG): container finished" podID="bd12f693-9299-40f2-aa8a-ee3afc279807" containerID="8c23b8e292825375c0231d316959c072f6805f65a702038aad3dd157edad24ff" exitCode=0 Jul 30 10:53:30.983737 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:30.983382 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" event={"ID":"bd12f693-9299-40f2-aa8a-ee3afc279807","Type":"ContainerDied","Data":"8c23b8e292825375c0231d316959c072f6805f65a702038aad3dd157edad24ff"} Jul 30 10:53:33.369714 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:53:33.369659 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd12f693_9299_40f2_aa8a_ee3afc279807.slice/crio-conmon-db4e495f54a309690664bcb98ec087a0e3f9f724662db63415b8a4c071ed812e.scope\": RecentStats: unable to find data in memory cache]" Jul 30 10:53:33.996778 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:33.996741 2575 generic.go:356] "Generic (PLEG): container finished" podID="bd12f693-9299-40f2-aa8a-ee3afc279807" containerID="db4e495f54a309690664bcb98ec087a0e3f9f724662db63415b8a4c071ed812e" exitCode=0 Jul 30 10:53:33.996997 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:33.996801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" event={"ID":"bd12f693-9299-40f2-aa8a-ee3afc279807","Type":"ContainerDied","Data":"db4e495f54a309690664bcb98ec087a0e3f9f724662db63415b8a4c071ed812e"} Jul 30 10:53:41.029958 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:41.029927 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" event={"ID":"bd12f693-9299-40f2-aa8a-ee3afc279807","Type":"ContainerStarted","Data":"b38d8d90f3b5948100b473015b866a9b7078d7298e2cddc2afa7e177f552ce7e"} Jul 30 10:53:41.048777 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:41.048728 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" podStartSLOduration=2.393932278 podStartE2EDuration="18.048712659s" totalImagesPullingTime="15.654780381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:53:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:53:41.04784565 +0000 UTC m=+329.020706684" watchObservedRunningTime="2026-07-30 10:53:41.048712659 +0000 UTC m=+329.021573693" Jul 30 10:53:42.037309 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:42.037272 2575 generic.go:356] "Generic (PLEG): container finished" podID="bd12f693-9299-40f2-aa8a-ee3afc279807" containerID="b38d8d90f3b5948100b473015b866a9b7078d7298e2cddc2afa7e177f552ce7e" exitCode=0 Jul 30 10:53:42.037687 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:42.037354 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" event={"ID":"bd12f693-9299-40f2-aa8a-ee3afc279807","Type":"ContainerDied","Data":"b38d8d90f3b5948100b473015b866a9b7078d7298e2cddc2afa7e177f552ce7e"} Jul 30 10:53:43.168633 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:43.168602 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:43.230150 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:43.230111 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-util\") pod \"bd12f693-9299-40f2-aa8a-ee3afc279807\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " Jul 30 10:53:43.230317 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:43.230177 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-bundle\") pod \"bd12f693-9299-40f2-aa8a-ee3afc279807\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " Jul 30 10:53:43.230317 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:43.230205 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk8cl\" (UniqueName: \"kubernetes.io/projected/bd12f693-9299-40f2-aa8a-ee3afc279807-kube-api-access-dk8cl\") pod \"bd12f693-9299-40f2-aa8a-ee3afc279807\" (UID: \"bd12f693-9299-40f2-aa8a-ee3afc279807\") " Jul 30 10:53:43.230963 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:43.230932 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-bundle" (OuterVolumeSpecName: "bundle") pod "bd12f693-9299-40f2-aa8a-ee3afc279807" (UID: "bd12f693-9299-40f2-aa8a-ee3afc279807"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:53:43.232409 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:43.232387 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd12f693-9299-40f2-aa8a-ee3afc279807-kube-api-access-dk8cl" (OuterVolumeSpecName: "kube-api-access-dk8cl") pod "bd12f693-9299-40f2-aa8a-ee3afc279807" (UID: "bd12f693-9299-40f2-aa8a-ee3afc279807"). InnerVolumeSpecName "kube-api-access-dk8cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:53:43.236666 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:43.236638 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-util" (OuterVolumeSpecName: "util") pod "bd12f693-9299-40f2-aa8a-ee3afc279807" (UID: "bd12f693-9299-40f2-aa8a-ee3afc279807"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:53:43.331726 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:43.331640 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-util\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:53:43.331726 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:43.331673 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd12f693-9299-40f2-aa8a-ee3afc279807-bundle\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:53:43.331726 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:43.331682 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk8cl\" (UniqueName: \"kubernetes.io/projected/bd12f693-9299-40f2-aa8a-ee3afc279807-kube-api-access-dk8cl\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:53:44.047605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:44.047579 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" Jul 30 10:53:44.047605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:44.047577 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540drd4t" event={"ID":"bd12f693-9299-40f2-aa8a-ee3afc279807","Type":"ContainerDied","Data":"a3a1a9cbac0f03accb7f3a5fada0eefb30c55d751ea40c794b2ce2b4a14f6442"} Jul 30 10:53:44.047815 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:44.047625 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a1a9cbac0f03accb7f3a5fada0eefb30c55d751ea40c794b2ce2b4a14f6442" Jul 30 10:53:51.209471 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.209384 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k"] Jul 30 10:53:51.209911 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.209867 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd12f693-9299-40f2-aa8a-ee3afc279807" containerName="extract" Jul 30 10:53:51.209911 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.209885 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd12f693-9299-40f2-aa8a-ee3afc279807" containerName="extract" Jul 30 10:53:51.209911 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.209909 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd12f693-9299-40f2-aa8a-ee3afc279807" containerName="pull" Jul 30 10:53:51.210068 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.209918 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd12f693-9299-40f2-aa8a-ee3afc279807" containerName="pull" Jul 30 10:53:51.210068 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.209939 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd12f693-9299-40f2-aa8a-ee3afc279807" containerName="util" Jul 30 10:53:51.210068 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.209949 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd12f693-9299-40f2-aa8a-ee3afc279807" containerName="util" Jul 30 10:53:51.210068 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.210027 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd12f693-9299-40f2-aa8a-ee3afc279807" containerName="extract" Jul 30 10:53:51.213444 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.213421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" Jul 30 10:53:51.215585 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.215556 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 30 10:53:51.215798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.215594 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hv79w\"" Jul 30 10:53:51.215798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.215679 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 30 10:53:51.226357 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.226323 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k"] Jul 30 10:53:51.297136 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.297067 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d234f335-144d-4987-9636-33211fa3bf1e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-plb6k\" (UID: \"d234f335-144d-4987-9636-33211fa3bf1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" Jul 30 10:53:51.297316 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.297236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm95k\" (UniqueName: \"kubernetes.io/projected/d234f335-144d-4987-9636-33211fa3bf1e-kube-api-access-cm95k\") pod \"cert-manager-operator-controller-manager-55c444bf86-plb6k\" (UID: \"d234f335-144d-4987-9636-33211fa3bf1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" Jul 30 10:53:51.397768 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.397722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm95k\" (UniqueName: \"kubernetes.io/projected/d234f335-144d-4987-9636-33211fa3bf1e-kube-api-access-cm95k\") pod \"cert-manager-operator-controller-manager-55c444bf86-plb6k\" (UID: \"d234f335-144d-4987-9636-33211fa3bf1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" Jul 30 10:53:51.397957 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.397783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d234f335-144d-4987-9636-33211fa3bf1e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-plb6k\" (UID: \"d234f335-144d-4987-9636-33211fa3bf1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" Jul 30 10:53:51.398253 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.398231 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d234f335-144d-4987-9636-33211fa3bf1e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-plb6k\" (UID: \"d234f335-144d-4987-9636-33211fa3bf1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" Jul 30 10:53:51.405990 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.405963 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm95k\" (UniqueName: \"kubernetes.io/projected/d234f335-144d-4987-9636-33211fa3bf1e-kube-api-access-cm95k\") pod \"cert-manager-operator-controller-manager-55c444bf86-plb6k\" (UID: \"d234f335-144d-4987-9636-33211fa3bf1e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" Jul 30 10:53:51.523353 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.523253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" Jul 30 10:53:51.660684 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:51.660591 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k"] Jul 30 10:53:51.663268 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:53:51.663238 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd234f335_144d_4987_9636_33211fa3bf1e.slice/crio-25eccc8058653237d00cb1fdeaf2cd68527386fd31e3a225ea98ce680f31c15a WatchSource:0}: Error finding container 25eccc8058653237d00cb1fdeaf2cd68527386fd31e3a225ea98ce680f31c15a: Status 404 returned error can't find the container with id 25eccc8058653237d00cb1fdeaf2cd68527386fd31e3a225ea98ce680f31c15a Jul 30 10:53:52.081300 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:52.081265 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" event={"ID":"d234f335-144d-4987-9636-33211fa3bf1e","Type":"ContainerStarted","Data":"25eccc8058653237d00cb1fdeaf2cd68527386fd31e3a225ea98ce680f31c15a"} Jul 30 10:53:54.092582 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:54.092547 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" event={"ID":"d234f335-144d-4987-9636-33211fa3bf1e","Type":"ContainerStarted","Data":"91ec77baf7163a6b3185c671e9e136c15ac9b4db4580c401c6df81664243bbfb"} Jul 30 10:53:54.112940 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:54.112891 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-plb6k" podStartSLOduration=0.782376176 podStartE2EDuration="3.112874673s" totalImagesPullingTime="2.330498497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:53:54.112340388 +0000 UTC m=+342.085201419" watchObservedRunningTime="2026-07-30 10:53:54.112874673 +0000 UTC m=+342.085735708" Jul 30 10:53:55.396678 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.396642 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54"] Jul 30 10:53:55.399174 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.399152 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.401379 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.401356 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ldxh6\"" Jul 30 10:53:55.432422 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.432395 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 10:53:55.432587 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.432399 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 10:53:55.434060 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.434041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54"] Jul 30 10:53:55.532620 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.532584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.532620 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.532627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.532887 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.532673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6k2\" (UniqueName: \"kubernetes.io/projected/45199a56-ab65-46e5-9463-2ad81c9253a6-kube-api-access-ch6k2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.633141 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.633081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6k2\" (UniqueName: \"kubernetes.io/projected/45199a56-ab65-46e5-9463-2ad81c9253a6-kube-api-access-ch6k2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.633305 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.633179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.633305 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.633210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.633563 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.633545 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.633599 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.633568 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.642667 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.642632 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6k2\" (UniqueName: \"kubernetes.io/projected/45199a56-ab65-46e5-9463-2ad81c9253a6-kube-api-access-ch6k2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.708645 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.708558 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:53:55.846426 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:55.846390 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54"] Jul 30 10:53:55.849270 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:53:55.849241 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45199a56_ab65_46e5_9463_2ad81c9253a6.slice/crio-11ee48461ad92fff5cd4679ab59da3954df6c174ef3b3815b8f7f41c53f2590c WatchSource:0}: Error finding container 11ee48461ad92fff5cd4679ab59da3954df6c174ef3b3815b8f7f41c53f2590c: Status 404 returned error can't find the container with id 11ee48461ad92fff5cd4679ab59da3954df6c174ef3b3815b8f7f41c53f2590c Jul 30 10:53:56.102940 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.102901 2575 generic.go:356] "Generic (PLEG): container finished" podID="45199a56-ab65-46e5-9463-2ad81c9253a6" containerID="c306c4e93ca5e9b85eb879f0b845498d2b79138d32d8e82b55dd3e4d68b6b0db" exitCode=0 Jul 30 10:53:56.103153 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.102988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" event={"ID":"45199a56-ab65-46e5-9463-2ad81c9253a6","Type":"ContainerDied","Data":"c306c4e93ca5e9b85eb879f0b845498d2b79138d32d8e82b55dd3e4d68b6b0db"} Jul 30 10:53:56.103153 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.103037 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" event={"ID":"45199a56-ab65-46e5-9463-2ad81c9253a6","Type":"ContainerStarted","Data":"11ee48461ad92fff5cd4679ab59da3954df6c174ef3b3815b8f7f41c53f2590c"} Jul 30 10:53:56.174487 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.174456 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gxhxw"] Jul 30 10:53:56.176762 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.176745 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" Jul 30 10:53:56.178574 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.178552 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 30 10:53:56.178686 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.178648 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-dlg8g\"" Jul 30 10:53:56.178813 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.178799 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 30 10:53:56.185587 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.185568 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gxhxw"] Jul 30 10:53:56.237419 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.237379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944mw\" (UniqueName: \"kubernetes.io/projected/86116d5f-0587-40e1-821c-57fca0d7c8a9-kube-api-access-944mw\") pod \"cert-manager-webhook-6486ffd997-gxhxw\" (UID: \"86116d5f-0587-40e1-821c-57fca0d7c8a9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" Jul 30 10:53:56.237419 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.237423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86116d5f-0587-40e1-821c-57fca0d7c8a9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gxhxw\" (UID: \"86116d5f-0587-40e1-821c-57fca0d7c8a9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" Jul 30 10:53:56.338462 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.338412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944mw\" (UniqueName: \"kubernetes.io/projected/86116d5f-0587-40e1-821c-57fca0d7c8a9-kube-api-access-944mw\") pod \"cert-manager-webhook-6486ffd997-gxhxw\" (UID: \"86116d5f-0587-40e1-821c-57fca0d7c8a9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" Jul 30 10:53:56.338462 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.338464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86116d5f-0587-40e1-821c-57fca0d7c8a9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gxhxw\" (UID: \"86116d5f-0587-40e1-821c-57fca0d7c8a9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" Jul 30 10:53:56.347256 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.347230 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86116d5f-0587-40e1-821c-57fca0d7c8a9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gxhxw\" (UID: \"86116d5f-0587-40e1-821c-57fca0d7c8a9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" Jul 30 10:53:56.347388 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.347307 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-944mw\" (UniqueName: \"kubernetes.io/projected/86116d5f-0587-40e1-821c-57fca0d7c8a9-kube-api-access-944mw\") pod \"cert-manager-webhook-6486ffd997-gxhxw\" (UID: \"86116d5f-0587-40e1-821c-57fca0d7c8a9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" Jul 30 10:53:56.492448 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.492345 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" Jul 30 10:53:56.623448 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:56.623352 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gxhxw"] Jul 30 10:53:56.625692 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:53:56.625667 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86116d5f_0587_40e1_821c_57fca0d7c8a9.slice/crio-018ff8a3175f61bef4c0d898620fcdf5a658e680f8268b8510dc89adbbc34699 WatchSource:0}: Error finding container 018ff8a3175f61bef4c0d898620fcdf5a658e680f8268b8510dc89adbbc34699: Status 404 returned error can't find the container with id 018ff8a3175f61bef4c0d898620fcdf5a658e680f8268b8510dc89adbbc34699 Jul 30 10:53:57.108728 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:53:57.108692 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" event={"ID":"86116d5f-0587-40e1-821c-57fca0d7c8a9","Type":"ContainerStarted","Data":"018ff8a3175f61bef4c0d898620fcdf5a658e680f8268b8510dc89adbbc34699"} Jul 30 10:54:03.141473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.141414 2575 generic.go:356] "Generic (PLEG): container finished" podID="45199a56-ab65-46e5-9463-2ad81c9253a6" containerID="953c38678f71bbbd5e79375e54a3493125c34cc7dc5d6466410219b386ce5c2b" exitCode=0 Jul 30 10:54:03.141954 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.141501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" event={"ID":"45199a56-ab65-46e5-9463-2ad81c9253a6","Type":"ContainerDied","Data":"953c38678f71bbbd5e79375e54a3493125c34cc7dc5d6466410219b386ce5c2b"} Jul 30 10:54:03.375189 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.375154 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-58v8d"] Jul 30 10:54:03.377748 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.377725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" Jul 30 10:54:03.379786 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.379762 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7bjvs\"" Jul 30 10:54:03.386235 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.385945 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-58v8d"] Jul 30 10:54:03.506498 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.506359 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rmh\" (UniqueName: \"kubernetes.io/projected/3f9fcd53-1e55-46fd-93ed-c321e591289b-kube-api-access-h8rmh\") pod \"cert-manager-cainjector-559cdd8c87-58v8d\" (UID: \"3f9fcd53-1e55-46fd-93ed-c321e591289b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" Jul 30 10:54:03.506670 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.506504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9fcd53-1e55-46fd-93ed-c321e591289b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-58v8d\" (UID: \"3f9fcd53-1e55-46fd-93ed-c321e591289b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" Jul 30 10:54:03.607191 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.607153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rmh\" (UniqueName: \"kubernetes.io/projected/3f9fcd53-1e55-46fd-93ed-c321e591289b-kube-api-access-h8rmh\") pod \"cert-manager-cainjector-559cdd8c87-58v8d\" (UID: \"3f9fcd53-1e55-46fd-93ed-c321e591289b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" Jul 30 10:54:03.607345 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.607247 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9fcd53-1e55-46fd-93ed-c321e591289b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-58v8d\" (UID: \"3f9fcd53-1e55-46fd-93ed-c321e591289b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" Jul 30 10:54:03.614527 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.614496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f9fcd53-1e55-46fd-93ed-c321e591289b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-58v8d\" (UID: \"3f9fcd53-1e55-46fd-93ed-c321e591289b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" Jul 30 10:54:03.614657 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.614565 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rmh\" (UniqueName: \"kubernetes.io/projected/3f9fcd53-1e55-46fd-93ed-c321e591289b-kube-api-access-h8rmh\") pod \"cert-manager-cainjector-559cdd8c87-58v8d\" (UID: \"3f9fcd53-1e55-46fd-93ed-c321e591289b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" Jul 30 10:54:03.689941 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.689900 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" Jul 30 10:54:03.831617 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:03.831590 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-58v8d"] Jul 30 10:54:03.834123 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:54:03.834084 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f9fcd53_1e55_46fd_93ed_c321e591289b.slice/crio-76ad0d6baab885552c1e690f977bcac510a6aab5792d2ed1c53cea1dac77ea13 WatchSource:0}: Error finding container 76ad0d6baab885552c1e690f977bcac510a6aab5792d2ed1c53cea1dac77ea13: Status 404 returned error can't find the container with id 76ad0d6baab885552c1e690f977bcac510a6aab5792d2ed1c53cea1dac77ea13 Jul 30 10:54:04.149202 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:04.149108 2575 generic.go:356] "Generic (PLEG): container finished" podID="45199a56-ab65-46e5-9463-2ad81c9253a6" containerID="aaab7a83125fb24ce9a90a60e9fa091e6cd75e3201265140eba3f9b8d19f5efb" exitCode=0 Jul 30 10:54:04.149631 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:04.149201 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" event={"ID":"45199a56-ab65-46e5-9463-2ad81c9253a6","Type":"ContainerDied","Data":"aaab7a83125fb24ce9a90a60e9fa091e6cd75e3201265140eba3f9b8d19f5efb"} Jul 30 10:54:04.150952 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:04.150927 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" event={"ID":"3f9fcd53-1e55-46fd-93ed-c321e591289b","Type":"ContainerStarted","Data":"a5c2a375b970a6d73e1be4987379e8b31d66c08a61d8b57b9ff15a6260c992e6"} Jul 30 10:54:04.151102 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:04.150964 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" event={"ID":"3f9fcd53-1e55-46fd-93ed-c321e591289b","Type":"ContainerStarted","Data":"76ad0d6baab885552c1e690f977bcac510a6aab5792d2ed1c53cea1dac77ea13"} Jul 30 10:54:04.152603 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:04.152584 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" event={"ID":"86116d5f-0587-40e1-821c-57fca0d7c8a9","Type":"ContainerStarted","Data":"b802dc20cca2ac6695525fc8480b40152ad0cd0fc7993634360b0d9cdf077f69"} Jul 30 10:54:04.152712 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:04.152692 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" Jul 30 10:54:04.206873 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:04.206767 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-58v8d" podStartSLOduration=1.206746295 podStartE2EDuration="1.206746295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:04.192436477 +0000 UTC m=+352.165297513" watchObservedRunningTime="2026-07-30 10:54:04.206746295 +0000 UTC m=+352.179607330" Jul 30 10:54:04.207738 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:04.207702 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" podStartSLOduration=1.256473492 podStartE2EDuration="8.207690691s" totalImagesPullingTime="6.951217199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:04.206669365 +0000 UTC m=+352.179530401" watchObservedRunningTime="2026-07-30 10:54:04.207690691 +0000 UTC m=+352.180551726" Jul 30 10:54:05.291033 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:05.291005 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:54:05.424399 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:05.424299 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch6k2\" (UniqueName: \"kubernetes.io/projected/45199a56-ab65-46e5-9463-2ad81c9253a6-kube-api-access-ch6k2\") pod \"45199a56-ab65-46e5-9463-2ad81c9253a6\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " Jul 30 10:54:05.424399 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:05.424374 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-bundle\") pod \"45199a56-ab65-46e5-9463-2ad81c9253a6\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " Jul 30 10:54:05.424399 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:05.424401 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-util\") pod \"45199a56-ab65-46e5-9463-2ad81c9253a6\" (UID: \"45199a56-ab65-46e5-9463-2ad81c9253a6\") " Jul 30 10:54:05.424758 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:05.424734 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-bundle" (OuterVolumeSpecName: "bundle") pod "45199a56-ab65-46e5-9463-2ad81c9253a6" (UID: "45199a56-ab65-46e5-9463-2ad81c9253a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:54:05.426635 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:05.426602 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45199a56-ab65-46e5-9463-2ad81c9253a6-kube-api-access-ch6k2" (OuterVolumeSpecName: "kube-api-access-ch6k2") pod "45199a56-ab65-46e5-9463-2ad81c9253a6" (UID: "45199a56-ab65-46e5-9463-2ad81c9253a6"). InnerVolumeSpecName "kube-api-access-ch6k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:54:05.429288 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:05.429258 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-util" (OuterVolumeSpecName: "util") pod "45199a56-ab65-46e5-9463-2ad81c9253a6" (UID: "45199a56-ab65-46e5-9463-2ad81c9253a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:54:05.525497 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:05.525457 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6k2\" (UniqueName: \"kubernetes.io/projected/45199a56-ab65-46e5-9463-2ad81c9253a6-kube-api-access-ch6k2\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:05.525497 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:05.525487 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-bundle\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:05.525497 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:05.525497 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45199a56-ab65-46e5-9463-2ad81c9253a6-util\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:06.163162 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:06.163136 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" Jul 30 10:54:06.163340 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:06.163137 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fs7t54" event={"ID":"45199a56-ab65-46e5-9463-2ad81c9253a6","Type":"ContainerDied","Data":"11ee48461ad92fff5cd4679ab59da3954df6c174ef3b3815b8f7f41c53f2590c"} Jul 30 10:54:06.163340 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:06.163247 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11ee48461ad92fff5cd4679ab59da3954df6c174ef3b3815b8f7f41c53f2590c" Jul 30 10:54:10.159847 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:10.159814 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-gxhxw" Jul 30 10:54:15.556738 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.556698 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc"] Jul 30 10:54:15.557356 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.557206 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45199a56-ab65-46e5-9463-2ad81c9253a6" containerName="extract" Jul 30 10:54:15.557356 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.557227 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45199a56-ab65-46e5-9463-2ad81c9253a6" containerName="extract" Jul 30 10:54:15.557356 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.557241 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45199a56-ab65-46e5-9463-2ad81c9253a6" containerName="util" Jul 30 10:54:15.557356 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.557250 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45199a56-ab65-46e5-9463-2ad81c9253a6" containerName="util" Jul 30 10:54:15.557356 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.557271 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45199a56-ab65-46e5-9463-2ad81c9253a6" containerName="pull" Jul 30 10:54:15.557356 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.557280 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45199a56-ab65-46e5-9463-2ad81c9253a6" containerName="pull" Jul 30 10:54:15.557667 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.557365 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="45199a56-ab65-46e5-9463-2ad81c9253a6" containerName="extract" Jul 30 10:54:15.562343 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.562312 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:15.564891 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.564866 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 10:54:15.565188 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.565173 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ldxh6\"" Jul 30 10:54:15.565250 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.565183 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 10:54:15.568118 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.568075 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc"] Jul 30 10:54:15.717440 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.717405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7tc\" (UniqueName: \"kubernetes.io/projected/be90779b-1f6b-4590-8bb4-2d84c159f1c9-kube-api-access-qb7tc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:15.717638 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.717469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:15.717638 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.717493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:15.819001 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.818913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7tc\" (UniqueName: \"kubernetes.io/projected/be90779b-1f6b-4590-8bb4-2d84c159f1c9-kube-api-access-qb7tc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:15.819001 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.818968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:15.819001 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.818994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:15.819403 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.819386 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:15.819448 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.819416 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:15.826761 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.826737 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7tc\" (UniqueName: \"kubernetes.io/projected/be90779b-1f6b-4590-8bb4-2d84c159f1c9-kube-api-access-qb7tc\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:15.873901 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:15.873860 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:16.008001 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:16.007966 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc"] Jul 30 10:54:16.010963 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:54:16.010935 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe90779b_1f6b_4590_8bb4_2d84c159f1c9.slice/crio-6002cf68a444c1599f46cbe670ef0460efe97bcb56f0eec39a46b1739b74b1a4 WatchSource:0}: Error finding container 6002cf68a444c1599f46cbe670ef0460efe97bcb56f0eec39a46b1739b74b1a4: Status 404 returned error can't find the container with id 6002cf68a444c1599f46cbe670ef0460efe97bcb56f0eec39a46b1739b74b1a4 Jul 30 10:54:16.207743 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:16.207707 2575 generic.go:356] "Generic (PLEG): container finished" podID="be90779b-1f6b-4590-8bb4-2d84c159f1c9" containerID="c63b398a6c922794fa1c2a179cff926c4a5e117e40b1d496ed34de9d3fe3fa0c" exitCode=0 Jul 30 10:54:16.207911 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:16.207753 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" event={"ID":"be90779b-1f6b-4590-8bb4-2d84c159f1c9","Type":"ContainerDied","Data":"c63b398a6c922794fa1c2a179cff926c4a5e117e40b1d496ed34de9d3fe3fa0c"} Jul 30 10:54:16.207911 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:16.207799 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" event={"ID":"be90779b-1f6b-4590-8bb4-2d84c159f1c9","Type":"ContainerStarted","Data":"6002cf68a444c1599f46cbe670ef0460efe97bcb56f0eec39a46b1739b74b1a4"} Jul 30 10:54:17.214356 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:17.214324 2575 generic.go:356] "Generic (PLEG): container finished" podID="be90779b-1f6b-4590-8bb4-2d84c159f1c9" containerID="2cf7adf69e6f701145f890ce92c69c8a0b1f06f084c0b4655466df85a392b5f2" exitCode=0 Jul 30 10:54:17.214743 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:17.214395 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" event={"ID":"be90779b-1f6b-4590-8bb4-2d84c159f1c9","Type":"ContainerDied","Data":"2cf7adf69e6f701145f890ce92c69c8a0b1f06f084c0b4655466df85a392b5f2"} Jul 30 10:54:18.221567 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:18.221531 2575 generic.go:356] "Generic (PLEG): container finished" podID="be90779b-1f6b-4590-8bb4-2d84c159f1c9" containerID="bb5e509524d94ea5815a65ffee43db3bc923f41cea2336a0fc188138d912c710" exitCode=0 Jul 30 10:54:18.221962 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:18.221584 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" event={"ID":"be90779b-1f6b-4590-8bb4-2d84c159f1c9","Type":"ContainerDied","Data":"bb5e509524d94ea5815a65ffee43db3bc923f41cea2336a0fc188138d912c710"} Jul 30 10:54:19.360351 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:19.360324 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:19.444204 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:19.444169 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-bundle\") pod \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " Jul 30 10:54:19.444388 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:19.444234 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-util\") pod \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " Jul 30 10:54:19.444388 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:19.444265 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb7tc\" (UniqueName: \"kubernetes.io/projected/be90779b-1f6b-4590-8bb4-2d84c159f1c9-kube-api-access-qb7tc\") pod \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\" (UID: \"be90779b-1f6b-4590-8bb4-2d84c159f1c9\") " Jul 30 10:54:19.445012 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:19.444980 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-bundle" (OuterVolumeSpecName: "bundle") pod "be90779b-1f6b-4590-8bb4-2d84c159f1c9" (UID: "be90779b-1f6b-4590-8bb4-2d84c159f1c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:54:19.446481 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:19.446459 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be90779b-1f6b-4590-8bb4-2d84c159f1c9-kube-api-access-qb7tc" (OuterVolumeSpecName: "kube-api-access-qb7tc") pod "be90779b-1f6b-4590-8bb4-2d84c159f1c9" (UID: "be90779b-1f6b-4590-8bb4-2d84c159f1c9"). InnerVolumeSpecName "kube-api-access-qb7tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:54:19.450115 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:19.450074 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-util" (OuterVolumeSpecName: "util") pod "be90779b-1f6b-4590-8bb4-2d84c159f1c9" (UID: "be90779b-1f6b-4590-8bb4-2d84c159f1c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:54:19.545787 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:19.545746 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-util\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:19.545787 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:19.545781 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb7tc\" (UniqueName: \"kubernetes.io/projected/be90779b-1f6b-4590-8bb4-2d84c159f1c9-kube-api-access-qb7tc\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:19.545787 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:19.545790 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be90779b-1f6b-4590-8bb4-2d84c159f1c9-bundle\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:20.233800 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:20.233771 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" Jul 30 10:54:20.233800 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:20.233792 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5lxrc" event={"ID":"be90779b-1f6b-4590-8bb4-2d84c159f1c9","Type":"ContainerDied","Data":"6002cf68a444c1599f46cbe670ef0460efe97bcb56f0eec39a46b1739b74b1a4"} Jul 30 10:54:20.234010 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:20.233834 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6002cf68a444c1599f46cbe670ef0460efe97bcb56f0eec39a46b1739b74b1a4" Jul 30 10:54:29.150373 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.150344 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr"] Jul 30 10:54:29.150798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.150678 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be90779b-1f6b-4590-8bb4-2d84c159f1c9" containerName="extract" Jul 30 10:54:29.150798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.150689 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="be90779b-1f6b-4590-8bb4-2d84c159f1c9" containerName="extract" Jul 30 10:54:29.150798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.150704 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be90779b-1f6b-4590-8bb4-2d84c159f1c9" containerName="pull" Jul 30 10:54:29.150798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.150710 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="be90779b-1f6b-4590-8bb4-2d84c159f1c9" containerName="pull" Jul 30 10:54:29.150798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.150726 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be90779b-1f6b-4590-8bb4-2d84c159f1c9" containerName="util" Jul 30 10:54:29.150798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.150732 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="be90779b-1f6b-4590-8bb4-2d84c159f1c9" containerName="util" Jul 30 10:54:29.150798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.150781 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="be90779b-1f6b-4590-8bb4-2d84c159f1c9" containerName="extract" Jul 30 10:54:29.153406 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.153382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.155420 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.155397 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 30 10:54:29.155532 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.155444 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 30 10:54:29.155837 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.155819 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 30 10:54:29.156157 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.156130 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wx6nc\"" Jul 30 10:54:29.156428 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.156414 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 30 10:54:29.168065 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.168041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr"] Jul 30 10:54:29.229541 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.229505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cebd5340-08de-4b45-aae8-01a1ee99ab76-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.229541 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.229539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvk7\" (UniqueName: \"kubernetes.io/projected/cebd5340-08de-4b45-aae8-01a1ee99ab76-kube-api-access-wmvk7\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.229812 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.229566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cebd5340-08de-4b45-aae8-01a1ee99ab76-tmp\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.229812 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.229617 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cebd5340-08de-4b45-aae8-01a1ee99ab76-manifests\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.229812 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.229661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cebd5340-08de-4b45-aae8-01a1ee99ab76-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.330817 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.330780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cebd5340-08de-4b45-aae8-01a1ee99ab76-tmp\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.331041 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.330826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cebd5340-08de-4b45-aae8-01a1ee99ab76-manifests\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.331041 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.330851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cebd5340-08de-4b45-aae8-01a1ee99ab76-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.331041 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.330898 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cebd5340-08de-4b45-aae8-01a1ee99ab76-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.331041 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.330920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvk7\" (UniqueName: \"kubernetes.io/projected/cebd5340-08de-4b45-aae8-01a1ee99ab76-kube-api-access-wmvk7\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.331388 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.331367 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cebd5340-08de-4b45-aae8-01a1ee99ab76-manifests\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.333356 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.333328 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cebd5340-08de-4b45-aae8-01a1ee99ab76-tmp\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.333484 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.333465 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cebd5340-08de-4b45-aae8-01a1ee99ab76-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.333563 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.333544 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cebd5340-08de-4b45-aae8-01a1ee99ab76-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.353164 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.353135 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvk7\" (UniqueName: \"kubernetes.io/projected/cebd5340-08de-4b45-aae8-01a1ee99ab76-kube-api-access-wmvk7\") pod \"opendatahub-operator-controller-manager-6d97895fdd-r8wjr\" (UID: \"cebd5340-08de-4b45-aae8-01a1ee99ab76\") " pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.464317 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.464216 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:29.610170 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:29.610136 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr"] Jul 30 10:54:29.613579 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:54:29.613556 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebd5340_08de_4b45_aae8_01a1ee99ab76.slice/crio-858ede5ee1eacd51df96a157db90ad9d92f934d381e917f7cfb0d1ad32ec1f34 WatchSource:0}: Error finding container 858ede5ee1eacd51df96a157db90ad9d92f934d381e917f7cfb0d1ad32ec1f34: Status 404 returned error can't find the container with id 858ede5ee1eacd51df96a157db90ad9d92f934d381e917f7cfb0d1ad32ec1f34 Jul 30 10:54:30.280872 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:30.280830 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" event={"ID":"cebd5340-08de-4b45-aae8-01a1ee99ab76","Type":"ContainerStarted","Data":"858ede5ee1eacd51df96a157db90ad9d92f934d381e917f7cfb0d1ad32ec1f34"} Jul 30 10:54:33.299284 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:33.299250 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" event={"ID":"cebd5340-08de-4b45-aae8-01a1ee99ab76","Type":"ContainerStarted","Data":"989970a39c707c007d26004cc76138fdaab4bef3310ac0330a9e0296c273c2fd"} Jul 30 10:54:34.306106 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:34.306059 2575 generic.go:356] "Generic (PLEG): container finished" podID="cebd5340-08de-4b45-aae8-01a1ee99ab76" containerID="989970a39c707c007d26004cc76138fdaab4bef3310ac0330a9e0296c273c2fd" exitCode=0 Jul 30 10:54:34.306509 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:34.306148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" event={"ID":"cebd5340-08de-4b45-aae8-01a1ee99ab76","Type":"ContainerDied","Data":"989970a39c707c007d26004cc76138fdaab4bef3310ac0330a9e0296c273c2fd"} Jul 30 10:54:35.313576 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:35.313541 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" event={"ID":"cebd5340-08de-4b45-aae8-01a1ee99ab76","Type":"ContainerStarted","Data":"51ffe4515a5e9491335b45ef9a8f791ba24403001276a09eea19414733c682b5"} Jul 30 10:54:35.313960 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:35.313761 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:35.334697 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:35.334640 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" podStartSLOduration=1.761803094 podStartE2EDuration="6.334618383s" totalImagesPullingTime="3.438569011s" totalInitContainerRuntime="1.134246278s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:54:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:35.333363929 +0000 UTC m=+383.306224966" watchObservedRunningTime="2026-07-30 10:54:35.334618383 +0000 UTC m=+383.307479418" Jul 30 10:54:46.319967 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:46.319937 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d97895fdd-r8wjr" Jul 30 10:54:48.625425 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.625391 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8"] Jul 30 10:54:48.635923 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.635758 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:48.638017 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.637989 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 30 10:54:48.638276 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.638255 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8"] Jul 30 10:54:48.638370 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.638320 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 30 10:54:48.638772 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.638755 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k8twm\"" Jul 30 10:54:48.710081 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.710031 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe82e77-f51a-452c-9213-0e779ce306ca-tls-certs\") pod \"kube-auth-proxy-8674569cb6-nlpn8\" (UID: \"4fe82e77-f51a-452c-9213-0e779ce306ca\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:48.710297 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.710117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqlp\" (UniqueName: \"kubernetes.io/projected/4fe82e77-f51a-452c-9213-0e779ce306ca-kube-api-access-rhqlp\") pod \"kube-auth-proxy-8674569cb6-nlpn8\" (UID: \"4fe82e77-f51a-452c-9213-0e779ce306ca\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:48.710297 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.710172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e77-f51a-452c-9213-0e779ce306ca-tmp\") pod \"kube-auth-proxy-8674569cb6-nlpn8\" (UID: \"4fe82e77-f51a-452c-9213-0e779ce306ca\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:48.711810 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.711776 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7f7f8"] Jul 30 10:54:48.714322 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.714299 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:54:48.716405 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.716385 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fklqn\"" Jul 30 10:54:48.716516 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.716385 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 30 10:54:48.724662 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.724602 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7f7f8"] Jul 30 10:54:48.811602 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.811569 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrdc\" (UniqueName: \"kubernetes.io/projected/5241e5b6-69c3-42e2-b691-3ead22d1e54a-kube-api-access-fgrdc\") pod \"odh-model-controller-7998c67dc4-7f7f8\" (UID: \"5241e5b6-69c3-42e2-b691-3ead22d1e54a\") " pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:54:48.811602 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.811612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe82e77-f51a-452c-9213-0e779ce306ca-tls-certs\") pod \"kube-auth-proxy-8674569cb6-nlpn8\" (UID: \"4fe82e77-f51a-452c-9213-0e779ce306ca\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:48.811838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.811637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqlp\" (UniqueName: \"kubernetes.io/projected/4fe82e77-f51a-452c-9213-0e779ce306ca-kube-api-access-rhqlp\") pod \"kube-auth-proxy-8674569cb6-nlpn8\" (UID: \"4fe82e77-f51a-452c-9213-0e779ce306ca\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:48.811838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.811659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e77-f51a-452c-9213-0e779ce306ca-tmp\") pod \"kube-auth-proxy-8674569cb6-nlpn8\" (UID: \"4fe82e77-f51a-452c-9213-0e779ce306ca\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:48.811838 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:54:48.811744 2575 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 30 10:54:48.811838 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:54:48.811805 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fe82e77-f51a-452c-9213-0e779ce306ca-tls-certs podName:4fe82e77-f51a-452c-9213-0e779ce306ca nodeName:}" failed. No retries permitted until 2026-07-30 10:54:49.311789455 +0000 UTC m=+397.284650471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4fe82e77-f51a-452c-9213-0e779ce306ca-tls-certs") pod "kube-auth-proxy-8674569cb6-nlpn8" (UID: "4fe82e77-f51a-452c-9213-0e779ce306ca") : secret "kube-auth-proxy-tls" not found Jul 30 10:54:48.811838 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.811748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5241e5b6-69c3-42e2-b691-3ead22d1e54a-cert\") pod \"odh-model-controller-7998c67dc4-7f7f8\" (UID: \"5241e5b6-69c3-42e2-b691-3ead22d1e54a\") " pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:54:48.813947 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.813929 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fe82e77-f51a-452c-9213-0e779ce306ca-tmp\") pod \"kube-auth-proxy-8674569cb6-nlpn8\" (UID: \"4fe82e77-f51a-452c-9213-0e779ce306ca\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:48.820273 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.820244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqlp\" (UniqueName: \"kubernetes.io/projected/4fe82e77-f51a-452c-9213-0e779ce306ca-kube-api-access-rhqlp\") pod \"kube-auth-proxy-8674569cb6-nlpn8\" (UID: \"4fe82e77-f51a-452c-9213-0e779ce306ca\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:48.913296 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.913212 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5241e5b6-69c3-42e2-b691-3ead22d1e54a-cert\") pod \"odh-model-controller-7998c67dc4-7f7f8\" (UID: \"5241e5b6-69c3-42e2-b691-3ead22d1e54a\") " pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:54:48.913296 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.913294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrdc\" (UniqueName: \"kubernetes.io/projected/5241e5b6-69c3-42e2-b691-3ead22d1e54a-kube-api-access-fgrdc\") pod \"odh-model-controller-7998c67dc4-7f7f8\" (UID: \"5241e5b6-69c3-42e2-b691-3ead22d1e54a\") " pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:54:48.913529 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:54:48.913360 2575 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 30 10:54:48.913529 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:54:48.913435 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5241e5b6-69c3-42e2-b691-3ead22d1e54a-cert podName:5241e5b6-69c3-42e2-b691-3ead22d1e54a nodeName:}" failed. No retries permitted until 2026-07-30 10:54:49.413415942 +0000 UTC m=+397.386276959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5241e5b6-69c3-42e2-b691-3ead22d1e54a-cert") pod "odh-model-controller-7998c67dc4-7f7f8" (UID: "5241e5b6-69c3-42e2-b691-3ead22d1e54a") : secret "odh-model-controller-webhook-cert" not found Jul 30 10:54:48.921863 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:48.921830 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrdc\" (UniqueName: \"kubernetes.io/projected/5241e5b6-69c3-42e2-b691-3ead22d1e54a-kube-api-access-fgrdc\") pod \"odh-model-controller-7998c67dc4-7f7f8\" (UID: \"5241e5b6-69c3-42e2-b691-3ead22d1e54a\") " pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:54:49.317798 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.317757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe82e77-f51a-452c-9213-0e779ce306ca-tls-certs\") pod \"kube-auth-proxy-8674569cb6-nlpn8\" (UID: \"4fe82e77-f51a-452c-9213-0e779ce306ca\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:49.320323 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.320298 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe82e77-f51a-452c-9213-0e779ce306ca-tls-certs\") pod \"kube-auth-proxy-8674569cb6-nlpn8\" (UID: \"4fe82e77-f51a-452c-9213-0e779ce306ca\") " pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:49.418770 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.418731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5241e5b6-69c3-42e2-b691-3ead22d1e54a-cert\") pod \"odh-model-controller-7998c67dc4-7f7f8\" (UID: \"5241e5b6-69c3-42e2-b691-3ead22d1e54a\") " pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:54:49.421221 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.421196 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5241e5b6-69c3-42e2-b691-3ead22d1e54a-cert\") pod \"odh-model-controller-7998c67dc4-7f7f8\" (UID: \"5241e5b6-69c3-42e2-b691-3ead22d1e54a\") " pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:54:49.546331 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.546290 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" Jul 30 10:54:49.625234 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.625009 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:54:49.634221 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.634193 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch"] Jul 30 10:54:49.636851 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.636830 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.639032 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.638968 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 30 10:54:49.639032 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.638980 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 30 10:54:49.639320 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.639302 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 30 10:54:49.639417 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.639343 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 30 10:54:49.639417 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.639392 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dmgt2\"" Jul 30 10:54:49.652822 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.652792 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch"] Jul 30 10:54:49.699249 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.699036 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8"] Jul 30 10:54:49.702062 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:54:49.702025 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe82e77_f51a_452c_9213_0e779ce306ca.slice/crio-a099e618922f3a698b613edf1bd425c6b94657c3126b49af6a91723847b65396 WatchSource:0}: Error finding container a099e618922f3a698b613edf1bd425c6b94657c3126b49af6a91723847b65396: Status 404 returned error can't find the container with id a099e618922f3a698b613edf1bd425c6b94657c3126b49af6a91723847b65396 Jul 30 10:54:49.722697 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.722669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.722847 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.722719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a92de7-b898-41ea-9670-8a3f0d2405e4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.722847 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.722763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52wtt\" (UniqueName: \"kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-kube-api-access-52wtt\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.722969 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.722881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.722969 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.722915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.722969 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.722945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.723144 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.722993 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.768692 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.768669 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7f7f8"] Jul 30 10:54:49.771267 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:54:49.771242 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5241e5b6_69c3_42e2_b691_3ead22d1e54a.slice/crio-2f389ebf2a477744e67a34d14ff7d0e73f4fd0b32f58f7a6286668bd5277ffd9 WatchSource:0}: Error finding container 2f389ebf2a477744e67a34d14ff7d0e73f4fd0b32f58f7a6286668bd5277ffd9: Status 404 returned error can't find the container with id 2f389ebf2a477744e67a34d14ff7d0e73f4fd0b32f58f7a6286668bd5277ffd9 Jul 30 10:54:49.824219 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.824188 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.824406 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.824237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.824406 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.824277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.824406 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.824307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a92de7-b898-41ea-9670-8a3f0d2405e4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.824406 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.824359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52wtt\" (UniqueName: \"kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-kube-api-access-52wtt\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.824590 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.824438 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.824590 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.824467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.824989 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.824961 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.827300 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.827269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.827431 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.827404 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.827558 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.827489 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a92de7-b898-41ea-9670-8a3f0d2405e4-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.827677 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.827659 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.831932 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.831903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52wtt\" (UniqueName: \"kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-kube-api-access-52wtt\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.832365 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.832341 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-48bch\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:49.949265 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:49.949177 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:50.092373 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:50.092345 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch"] Jul 30 10:54:50.094234 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:54:50.094208 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a92de7_b898_41ea_9670_8a3f0d2405e4.slice/crio-8a9691b1159e8fa59428edba6eaffb1394e647fa3df6ce7094d0b4e41ed332ea WatchSource:0}: Error finding container 8a9691b1159e8fa59428edba6eaffb1394e647fa3df6ce7094d0b4e41ed332ea: Status 404 returned error can't find the container with id 8a9691b1159e8fa59428edba6eaffb1394e647fa3df6ce7094d0b4e41ed332ea Jul 30 10:54:50.390210 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:50.390171 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" event={"ID":"5241e5b6-69c3-42e2-b691-3ead22d1e54a","Type":"ContainerStarted","Data":"2f389ebf2a477744e67a34d14ff7d0e73f4fd0b32f58f7a6286668bd5277ffd9"} Jul 30 10:54:50.391605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:50.391575 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" event={"ID":"4fe82e77-f51a-452c-9213-0e779ce306ca","Type":"ContainerStarted","Data":"a099e618922f3a698b613edf1bd425c6b94657c3126b49af6a91723847b65396"} Jul 30 10:54:50.393099 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:50.393055 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" event={"ID":"d5a92de7-b898-41ea-9670-8a3f0d2405e4","Type":"ContainerStarted","Data":"8a9691b1159e8fa59428edba6eaffb1394e647fa3df6ce7094d0b4e41ed332ea"} Jul 30 10:54:52.405166 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:52.405121 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" event={"ID":"4fe82e77-f51a-452c-9213-0e779ce306ca","Type":"ContainerStarted","Data":"face912297c2d6fb220d7efae8ef36379e5e8e2d905d341e5f725437d314d003"} Jul 30 10:54:52.423842 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:52.423777 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8674569cb6-nlpn8" podStartSLOduration=1.867373604 podStartE2EDuration="4.423758656s" totalImagesPullingTime="2.556385052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:52.421019842 +0000 UTC m=+400.393880897" watchObservedRunningTime="2026-07-30 10:54:52.423758656 +0000 UTC m=+400.396619691" Jul 30 10:54:55.398920 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:55.398876 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 10:54:55.399242 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:55.398961 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 10:54:56.338811 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.338773 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv"] Jul 30 10:54:56.342665 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.342647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.353335 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.353307 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv"] Jul 30 10:54:56.430664 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.430625 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" event={"ID":"d5a92de7-b898-41ea-9670-8a3f0d2405e4","Type":"ContainerStarted","Data":"15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e"} Jul 30 10:54:56.431151 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.430690 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:56.432309 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.432280 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" event={"ID":"5241e5b6-69c3-42e2-b691-3ead22d1e54a","Type":"ContainerStarted","Data":"24826fd9af06954664532286fc70b8f490db6b9d8a0ed5a0a7164192d9a10212"} Jul 30 10:54:56.432494 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.432460 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:54:56.432774 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.432751 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-48bch container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 10:54:56.432839 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.432805 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" podUID="d5a92de7-b898-41ea-9670-8a3f0d2405e4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 10:54:56.451155 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.451105 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" podStartSLOduration=2.14896301 podStartE2EDuration="7.451078046s" totalImagesPullingTime="5.302115036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:56.449698448 +0000 UTC m=+404.422559486" watchObservedRunningTime="2026-07-30 10:54:56.451078046 +0000 UTC m=+404.423939081" Jul 30 10:54:56.465217 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.465164 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" podStartSLOduration=2.839296457 podStartE2EDuration="8.46514739s" totalImagesPullingTime="5.625850933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:56.464580263 +0000 UTC m=+404.437441299" watchObservedRunningTime="2026-07-30 10:54:56.46514739 +0000 UTC m=+404.438008426" Jul 30 10:54:56.492303 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.492277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df10590c-6edd-48b3-a156-bf21353dacd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.492450 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.492317 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgv8\" (UniqueName: \"kubernetes.io/projected/df10590c-6edd-48b3-a156-bf21353dacd8-kube-api-access-pcgv8\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.492450 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.492350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df10590c-6edd-48b3-a156-bf21353dacd8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.492450 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.492438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df10590c-6edd-48b3-a156-bf21353dacd8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.492558 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.492479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df10590c-6edd-48b3-a156-bf21353dacd8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.492558 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.492528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df10590c-6edd-48b3-a156-bf21353dacd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.492635 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.492559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df10590c-6edd-48b3-a156-bf21353dacd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.594136 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.594032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df10590c-6edd-48b3-a156-bf21353dacd8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.594318 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.594131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df10590c-6edd-48b3-a156-bf21353dacd8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.594318 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.594187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df10590c-6edd-48b3-a156-bf21353dacd8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.594318 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.594243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df10590c-6edd-48b3-a156-bf21353dacd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.594490 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.594381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df10590c-6edd-48b3-a156-bf21353dacd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.594542 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.594494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df10590c-6edd-48b3-a156-bf21353dacd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.594593 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.594559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgv8\" (UniqueName: \"kubernetes.io/projected/df10590c-6edd-48b3-a156-bf21353dacd8-kube-api-access-pcgv8\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.595462 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.595416 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/df10590c-6edd-48b3-a156-bf21353dacd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.596932 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.596889 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/df10590c-6edd-48b3-a156-bf21353dacd8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.600330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.597061 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df10590c-6edd-48b3-a156-bf21353dacd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.600330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.597343 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/df10590c-6edd-48b3-a156-bf21353dacd8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.600330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.598549 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/df10590c-6edd-48b3-a156-bf21353dacd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.604717 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.604669 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df10590c-6edd-48b3-a156-bf21353dacd8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.604929 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.604721 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgv8\" (UniqueName: \"kubernetes.io/projected/df10590c-6edd-48b3-a156-bf21353dacd8-kube-api-access-pcgv8\") pod \"istiod-openshift-gateway-5677ccb49b-qtsjv\" (UID: \"df10590c-6edd-48b3-a156-bf21353dacd8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.654625 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.654598 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:56.837612 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.837583 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv"] Jul 30 10:54:56.838521 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:54:56.838495 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf10590c_6edd_48b3_a156_bf21353dacd8.slice/crio-1ceb458390cc5fb8650ea22af1408c83ecd034cd119acf71ad805af44146b2ed WatchSource:0}: Error finding container 1ceb458390cc5fb8650ea22af1408c83ecd034cd119acf71ad805af44146b2ed: Status 404 returned error can't find the container with id 1ceb458390cc5fb8650ea22af1408c83ecd034cd119acf71ad805af44146b2ed Jul 30 10:54:56.842249 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.842211 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 10:54:56.842326 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:56.842288 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 10:54:57.442343 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:57.442304 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" event={"ID":"df10590c-6edd-48b3-a156-bf21353dacd8","Type":"ContainerStarted","Data":"c483c1f7919bcd18ef215251889fb565f240fc92b8cbe0027aa34de70b17206d"} Jul 30 10:54:57.442884 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:57.442400 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" event={"ID":"df10590c-6edd-48b3-a156-bf21353dacd8","Type":"ContainerStarted","Data":"1ceb458390cc5fb8650ea22af1408c83ecd034cd119acf71ad805af44146b2ed"} Jul 30 10:54:57.442884 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:57.442665 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:57.443287 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:57.443237 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:57.444566 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:57.444541 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-qtsjv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 10:54:57.444652 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:57.444594 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" podUID="df10590c-6edd-48b3-a156-bf21353dacd8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 10:54:57.467294 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:57.467250 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" podStartSLOduration=1.46723502 podStartE2EDuration="1.46723502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:54:57.465924322 +0000 UTC m=+405.438785372" watchObservedRunningTime="2026-07-30 10:54:57.46723502 +0000 UTC m=+405.440096055" Jul 30 10:54:58.450975 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:58.450932 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-qtsjv" Jul 30 10:54:58.527815 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:58.527788 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch"] Jul 30 10:54:59.455316 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.455238 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" podUID="d5a92de7-b898-41ea-9670-8a3f0d2405e4" containerName="discovery" containerID="cri-o://15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e" gracePeriod=30 Jul 30 10:54:59.702937 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.702914 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:54:59.724491 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.724403 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-cacerts\") pod \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " Jul 30 10:54:59.724491 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.724445 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-dns-cert\") pod \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " Jul 30 10:54:59.724491 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.724467 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-kubeconfig\") pod \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " Jul 30 10:54:59.724758 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.724504 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a92de7-b898-41ea-9670-8a3f0d2405e4-local-certs\") pod \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " Jul 30 10:54:59.724758 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.724544 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52wtt\" (UniqueName: \"kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-kube-api-access-52wtt\") pod \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " Jul 30 10:54:59.724758 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.724712 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-token\") pod \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " Jul 30 10:54:59.724924 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.724789 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-ca-configmap\") pod \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\" (UID: \"d5a92de7-b898-41ea-9670-8a3f0d2405e4\") " Jul 30 10:54:59.725976 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.725943 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d5a92de7-b898-41ea-9670-8a3f0d2405e4" (UID: "d5a92de7-b898-41ea-9670-8a3f0d2405e4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 10:54:59.727660 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.727637 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d5a92de7-b898-41ea-9670-8a3f0d2405e4" (UID: "d5a92de7-b898-41ea-9670-8a3f0d2405e4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:54:59.728327 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.728277 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-kube-api-access-52wtt" (OuterVolumeSpecName: "kube-api-access-52wtt") pod "d5a92de7-b898-41ea-9670-8a3f0d2405e4" (UID: "d5a92de7-b898-41ea-9670-8a3f0d2405e4"). InnerVolumeSpecName "kube-api-access-52wtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:54:59.728487 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.728378 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-cacerts" (OuterVolumeSpecName: "cacerts") pod "d5a92de7-b898-41ea-9670-8a3f0d2405e4" (UID: "d5a92de7-b898-41ea-9670-8a3f0d2405e4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:54:59.728855 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.728824 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a92de7-b898-41ea-9670-8a3f0d2405e4-local-certs" (OuterVolumeSpecName: "local-certs") pod "d5a92de7-b898-41ea-9670-8a3f0d2405e4" (UID: "d5a92de7-b898-41ea-9670-8a3f0d2405e4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:54:59.728942 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.728890 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d5a92de7-b898-41ea-9670-8a3f0d2405e4" (UID: "d5a92de7-b898-41ea-9670-8a3f0d2405e4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:54:59.729057 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.729037 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5a92de7-b898-41ea-9670-8a3f0d2405e4" (UID: "d5a92de7-b898-41ea-9670-8a3f0d2405e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:54:59.825493 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.825458 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-kubeconfig\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:59.825493 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.825489 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a92de7-b898-41ea-9670-8a3f0d2405e4-local-certs\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:59.825493 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.825501 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52wtt\" (UniqueName: \"kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-kube-api-access-52wtt\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:59.825746 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.825509 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-token\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:59.825746 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.825518 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-ca-configmap\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:59.825746 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.825526 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-cacerts\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:54:59.825746 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:54:59.825534 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5a92de7-b898-41ea-9670-8a3f0d2405e4-istio-csr-dns-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:00.462382 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:00.462347 2575 generic.go:356] "Generic (PLEG): container finished" podID="d5a92de7-b898-41ea-9670-8a3f0d2405e4" containerID="15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e" exitCode=0 Jul 30 10:55:00.462871 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:00.462395 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" Jul 30 10:55:00.462871 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:00.462434 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" event={"ID":"d5a92de7-b898-41ea-9670-8a3f0d2405e4","Type":"ContainerDied","Data":"15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e"} Jul 30 10:55:00.462871 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:00.462485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch" event={"ID":"d5a92de7-b898-41ea-9670-8a3f0d2405e4","Type":"ContainerDied","Data":"8a9691b1159e8fa59428edba6eaffb1394e647fa3df6ce7094d0b4e41ed332ea"} Jul 30 10:55:00.462871 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:00.462508 2575 scope.go:117] "RemoveContainer" containerID="15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e" Jul 30 10:55:00.471959 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:00.471935 2575 scope.go:117] "RemoveContainer" containerID="15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e" Jul 30 10:55:00.472309 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:55:00.472291 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e\": container with ID starting with 15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e not found: ID does not exist" containerID="15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e" Jul 30 10:55:00.472427 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:00.472319 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e"} err="failed to get container status \"15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e\": rpc error: code = NotFound desc = could not find container \"15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e\": container with ID starting with 15d0e0d600fb97f6f70b7e077fce513ac0d2f696be1a81b00180e1884523748e not found: ID does not exist" Jul 30 10:55:00.487749 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:00.487713 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch"] Jul 30 10:55:00.497400 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:00.497367 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-48bch"] Jul 30 10:55:00.527233 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:00.527193 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a92de7-b898-41ea-9670-8a3f0d2405e4" path="/var/lib/kubelet/pods/d5a92de7-b898-41ea-9670-8a3f0d2405e4/volumes" Jul 30 10:55:07.444935 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:07.444894 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-7f7f8" Jul 30 10:55:52.440623 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.440586 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2"] Jul 30 10:55:52.441178 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.441142 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a92de7-b898-41ea-9670-8a3f0d2405e4" containerName="discovery" Jul 30 10:55:52.441178 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.441162 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a92de7-b898-41ea-9670-8a3f0d2405e4" containerName="discovery" Jul 30 10:55:52.441307 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.441281 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a92de7-b898-41ea-9670-8a3f0d2405e4" containerName="discovery" Jul 30 10:55:52.443379 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.443356 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.445285 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.445267 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 10:55:52.445387 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.445290 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 10:55:52.445675 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.445662 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ldxh6\"" Jul 30 10:55:52.451960 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.451935 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2"] Jul 30 10:55:52.499828 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.499790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.499828 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.499831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4jsk\" (UniqueName: \"kubernetes.io/projected/cf443429-dcd0-4405-94cf-d2718d49b287-kube-api-access-m4jsk\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.500044 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.499900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.536991 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.536960 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7"] Jul 30 10:55:52.539528 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.539507 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.548385 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.548345 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7"] Jul 30 10:55:52.600892 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.600855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.601067 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.600903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.601067 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.600931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.601067 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.600953 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn68r\" (UniqueName: \"kubernetes.io/projected/e118f86a-cac1-43fd-9e92-84d465cba441-kube-api-access-hn68r\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.601067 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.600991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.601067 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.601011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4jsk\" (UniqueName: \"kubernetes.io/projected/cf443429-dcd0-4405-94cf-d2718d49b287-kube-api-access-m4jsk\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.601367 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.601228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.601367 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.601310 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.608367 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.608340 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4jsk\" (UniqueName: \"kubernetes.io/projected/cf443429-dcd0-4405-94cf-d2718d49b287-kube-api-access-m4jsk\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.638070 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.638031 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf"] Jul 30 10:55:52.640890 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.640873 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.649632 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.649605 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf"] Jul 30 10:55:52.702329 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.702239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.702329 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.702286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.702566 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.702333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.702566 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.702369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn68r\" (UniqueName: \"kubernetes.io/projected/e118f86a-cac1-43fd-9e92-84d465cba441-kube-api-access-hn68r\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.702566 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.702425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sml2m\" (UniqueName: \"kubernetes.io/projected/15a7254a-c81f-416d-aeb3-38635e345123-kube-api-access-sml2m\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.702566 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.702450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.702780 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.702681 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.702780 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.702744 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.709904 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.709879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn68r\" (UniqueName: \"kubernetes.io/projected/e118f86a-cac1-43fd-9e92-84d465cba441-kube-api-access-hn68r\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.738487 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.738453 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz"] Jul 30 10:55:52.741138 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.741120 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:52.750070 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.750041 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz"] Jul 30 10:55:52.753122 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.753100 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:52.803654 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.803596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:52.803839 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.803660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sml2m\" (UniqueName: \"kubernetes.io/projected/15a7254a-c81f-416d-aeb3-38635e345123-kube-api-access-sml2m\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.803839 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.803691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.803839 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.803757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddpc\" (UniqueName: \"kubernetes.io/projected/f4501bd0-504b-43d9-8034-770a20df60c2-kube-api-access-cddpc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:52.803839 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.803795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:52.803839 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.803835 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.804388 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.804305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.804388 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.804349 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.812884 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.812813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sml2m\" (UniqueName: \"kubernetes.io/projected/15a7254a-c81f-416d-aeb3-38635e345123-kube-api-access-sml2m\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.849433 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.849401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:52.894394 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.894362 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2"] Jul 30 10:55:52.896656 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:55:52.896625 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf443429_dcd0_4405_94cf_d2718d49b287.slice/crio-f03d4daa7ba13fb6d3bc96ec60000013660cd323f9ea654cbae280149d003e57 WatchSource:0}: Error finding container f03d4daa7ba13fb6d3bc96ec60000013660cd323f9ea654cbae280149d003e57: Status 404 returned error can't find the container with id f03d4daa7ba13fb6d3bc96ec60000013660cd323f9ea654cbae280149d003e57 Jul 30 10:55:52.906933 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.906892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddpc\" (UniqueName: \"kubernetes.io/projected/f4501bd0-504b-43d9-8034-770a20df60c2-kube-api-access-cddpc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:52.907055 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.906946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:52.907055 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.907046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:52.907578 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.907555 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:52.907682 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.907654 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:52.916181 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.916149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddpc\" (UniqueName: \"kubernetes.io/projected/f4501bd0-504b-43d9-8034-770a20df60c2-kube-api-access-cddpc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:52.951364 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.951328 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:52.997265 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:52.997234 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7"] Jul 30 10:55:53.002062 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:55:53.002025 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode118f86a_cac1_43fd_9e92_84d465cba441.slice/crio-da2117faeeeaf264524e7b7a85faad140ba4fd55df4f03ea3649aa7fdc46272a WatchSource:0}: Error finding container da2117faeeeaf264524e7b7a85faad140ba4fd55df4f03ea3649aa7fdc46272a: Status 404 returned error can't find the container with id da2117faeeeaf264524e7b7a85faad140ba4fd55df4f03ea3649aa7fdc46272a Jul 30 10:55:53.052056 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.052020 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:53.107524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.107488 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf"] Jul 30 10:55:53.111165 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:55:53.111136 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a7254a_c81f_416d_aeb3_38635e345123.slice/crio-af81f4f49268095fa8c3effe1f7aedfdfbffdba2a10272a91f4d8d92110f6ac4 WatchSource:0}: Error finding container af81f4f49268095fa8c3effe1f7aedfdfbffdba2a10272a91f4d8d92110f6ac4: Status 404 returned error can't find the container with id af81f4f49268095fa8c3effe1f7aedfdfbffdba2a10272a91f4d8d92110f6ac4 Jul 30 10:55:53.202480 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.202452 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz"] Jul 30 10:55:53.210301 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:55:53.210274 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4501bd0_504b_43d9_8034_770a20df60c2.slice/crio-2ba5d0c9a425b7a45cb67f862891760fb8d16bd5fed54da6693a1c51cd37ac7e WatchSource:0}: Error finding container 2ba5d0c9a425b7a45cb67f862891760fb8d16bd5fed54da6693a1c51cd37ac7e: Status 404 returned error can't find the container with id 2ba5d0c9a425b7a45cb67f862891760fb8d16bd5fed54da6693a1c51cd37ac7e Jul 30 10:55:53.726886 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.726851 2575 generic.go:356] "Generic (PLEG): container finished" podID="f4501bd0-504b-43d9-8034-770a20df60c2" containerID="fa29b91ead4a5116177a5c4cbbaeb5bb1b0cae9b50c2601768e91483982bc69c" exitCode=0 Jul 30 10:55:53.727349 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.726918 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" event={"ID":"f4501bd0-504b-43d9-8034-770a20df60c2","Type":"ContainerDied","Data":"fa29b91ead4a5116177a5c4cbbaeb5bb1b0cae9b50c2601768e91483982bc69c"} Jul 30 10:55:53.727349 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.726971 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" event={"ID":"f4501bd0-504b-43d9-8034-770a20df60c2","Type":"ContainerStarted","Data":"2ba5d0c9a425b7a45cb67f862891760fb8d16bd5fed54da6693a1c51cd37ac7e"} Jul 30 10:55:53.728699 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.728611 2575 generic.go:356] "Generic (PLEG): container finished" podID="15a7254a-c81f-416d-aeb3-38635e345123" containerID="0c8e5213341761d729dcdb0dfe1cc158b390b2b049d4cc3e22dc420f0480a04c" exitCode=0 Jul 30 10:55:53.728699 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.728689 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" event={"ID":"15a7254a-c81f-416d-aeb3-38635e345123","Type":"ContainerDied","Data":"0c8e5213341761d729dcdb0dfe1cc158b390b2b049d4cc3e22dc420f0480a04c"} Jul 30 10:55:53.728823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.728733 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" event={"ID":"15a7254a-c81f-416d-aeb3-38635e345123","Type":"ContainerStarted","Data":"af81f4f49268095fa8c3effe1f7aedfdfbffdba2a10272a91f4d8d92110f6ac4"} Jul 30 10:55:53.730408 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.730383 2575 generic.go:356] "Generic (PLEG): container finished" podID="e118f86a-cac1-43fd-9e92-84d465cba441" containerID="8c5e79604dae1ba6b16c01d186a3c11b74c56f607e79eac5a263ab2edd6ffd9c" exitCode=0 Jul 30 10:55:53.730524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.730462 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" event={"ID":"e118f86a-cac1-43fd-9e92-84d465cba441","Type":"ContainerDied","Data":"8c5e79604dae1ba6b16c01d186a3c11b74c56f607e79eac5a263ab2edd6ffd9c"} Jul 30 10:55:53.730524 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.730495 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" event={"ID":"e118f86a-cac1-43fd-9e92-84d465cba441","Type":"ContainerStarted","Data":"da2117faeeeaf264524e7b7a85faad140ba4fd55df4f03ea3649aa7fdc46272a"} Jul 30 10:55:53.732553 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.732528 2575 generic.go:356] "Generic (PLEG): container finished" podID="cf443429-dcd0-4405-94cf-d2718d49b287" containerID="92a3dae98c7092f3bfb853869de2fb8a248aaefc9f21b761a9ae2dd0319187bf" exitCode=0 Jul 30 10:55:53.732668 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.732651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" event={"ID":"cf443429-dcd0-4405-94cf-d2718d49b287","Type":"ContainerDied","Data":"92a3dae98c7092f3bfb853869de2fb8a248aaefc9f21b761a9ae2dd0319187bf"} Jul 30 10:55:53.732768 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:53.732748 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" event={"ID":"cf443429-dcd0-4405-94cf-d2718d49b287","Type":"ContainerStarted","Data":"f03d4daa7ba13fb6d3bc96ec60000013660cd323f9ea654cbae280149d003e57"} Jul 30 10:55:54.740666 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:54.740634 2575 generic.go:356] "Generic (PLEG): container finished" podID="f4501bd0-504b-43d9-8034-770a20df60c2" containerID="25e3b7fc8de6aae2abd6dfba9fef65cfd518d2cfb3ded7900c01a4e5fcbc7712" exitCode=0 Jul 30 10:55:54.741105 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:54.740708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" event={"ID":"f4501bd0-504b-43d9-8034-770a20df60c2","Type":"ContainerDied","Data":"25e3b7fc8de6aae2abd6dfba9fef65cfd518d2cfb3ded7900c01a4e5fcbc7712"} Jul 30 10:55:54.742705 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:54.742683 2575 generic.go:356] "Generic (PLEG): container finished" podID="15a7254a-c81f-416d-aeb3-38635e345123" containerID="5e12a1f3ea69d633b79cc568e5f792b415488a7c56dee10192b9434a6ba96108" exitCode=0 Jul 30 10:55:54.742796 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:54.742783 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" event={"ID":"15a7254a-c81f-416d-aeb3-38635e345123","Type":"ContainerDied","Data":"5e12a1f3ea69d633b79cc568e5f792b415488a7c56dee10192b9434a6ba96108"} Jul 30 10:55:54.744880 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:54.744859 2575 generic.go:356] "Generic (PLEG): container finished" podID="cf443429-dcd0-4405-94cf-d2718d49b287" containerID="013e7d7f17e6ea5c11f5d21b21a09fe60a625b03480852de511ce93f77c2ee19" exitCode=0 Jul 30 10:55:54.744990 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:54.744899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" event={"ID":"cf443429-dcd0-4405-94cf-d2718d49b287","Type":"ContainerDied","Data":"013e7d7f17e6ea5c11f5d21b21a09fe60a625b03480852de511ce93f77c2ee19"} Jul 30 10:55:55.753459 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:55.753419 2575 generic.go:356] "Generic (PLEG): container finished" podID="f4501bd0-504b-43d9-8034-770a20df60c2" containerID="71aabe1c4b17c8903ea2fdfac2032e9c809058e5e934907f1b225127701d6a3b" exitCode=0 Jul 30 10:55:55.753850 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:55.753508 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" event={"ID":"f4501bd0-504b-43d9-8034-770a20df60c2","Type":"ContainerDied","Data":"71aabe1c4b17c8903ea2fdfac2032e9c809058e5e934907f1b225127701d6a3b"} Jul 30 10:55:55.755480 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:55.755457 2575 generic.go:356] "Generic (PLEG): container finished" podID="15a7254a-c81f-416d-aeb3-38635e345123" containerID="59e502d783c9666bd8b89ef44e506d70bec9b4af8dd5ca1343ae0b98227c7bb5" exitCode=0 Jul 30 10:55:55.755596 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:55.755536 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" event={"ID":"15a7254a-c81f-416d-aeb3-38635e345123","Type":"ContainerDied","Data":"59e502d783c9666bd8b89ef44e506d70bec9b4af8dd5ca1343ae0b98227c7bb5"} Jul 30 10:55:55.757505 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:55.757484 2575 generic.go:356] "Generic (PLEG): container finished" podID="cf443429-dcd0-4405-94cf-d2718d49b287" containerID="03e9260bcc5b14986446d9354094857e803c6a659246ca49fc3ba22a66e62ef7" exitCode=0 Jul 30 10:55:55.757596 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:55.757569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" event={"ID":"cf443429-dcd0-4405-94cf-d2718d49b287","Type":"ContainerDied","Data":"03e9260bcc5b14986446d9354094857e803c6a659246ca49fc3ba22a66e62ef7"} Jul 30 10:55:56.765251 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:56.765216 2575 generic.go:356] "Generic (PLEG): container finished" podID="e118f86a-cac1-43fd-9e92-84d465cba441" containerID="c1470ab4ca09288c8f4cfc9fe1391ee477fa19f322ee7b5efb15ecd1c13e084f" exitCode=0 Jul 30 10:55:56.765833 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:56.765308 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" event={"ID":"e118f86a-cac1-43fd-9e92-84d465cba441","Type":"ContainerDied","Data":"c1470ab4ca09288c8f4cfc9fe1391ee477fa19f322ee7b5efb15ecd1c13e084f"} Jul 30 10:55:56.921853 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:56.921821 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:56.995583 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:56.995558 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:56.998592 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:56.998574 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:57.049604 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.049575 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sml2m\" (UniqueName: \"kubernetes.io/projected/15a7254a-c81f-416d-aeb3-38635e345123-kube-api-access-sml2m\") pod \"15a7254a-c81f-416d-aeb3-38635e345123\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " Jul 30 10:55:57.049791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.049624 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-bundle\") pod \"15a7254a-c81f-416d-aeb3-38635e345123\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " Jul 30 10:55:57.049791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.049669 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cddpc\" (UniqueName: \"kubernetes.io/projected/f4501bd0-504b-43d9-8034-770a20df60c2-kube-api-access-cddpc\") pod \"f4501bd0-504b-43d9-8034-770a20df60c2\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " Jul 30 10:55:57.049791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.049690 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-util\") pod \"f4501bd0-504b-43d9-8034-770a20df60c2\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " Jul 30 10:55:57.049791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.049708 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-bundle\") pod \"cf443429-dcd0-4405-94cf-d2718d49b287\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " Jul 30 10:55:57.049791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.049743 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-bundle\") pod \"f4501bd0-504b-43d9-8034-770a20df60c2\" (UID: \"f4501bd0-504b-43d9-8034-770a20df60c2\") " Jul 30 10:55:57.049791 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.049787 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4jsk\" (UniqueName: \"kubernetes.io/projected/cf443429-dcd0-4405-94cf-d2718d49b287-kube-api-access-m4jsk\") pod \"cf443429-dcd0-4405-94cf-d2718d49b287\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " Jul 30 10:55:57.050143 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.049812 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-util\") pod \"cf443429-dcd0-4405-94cf-d2718d49b287\" (UID: \"cf443429-dcd0-4405-94cf-d2718d49b287\") " Jul 30 10:55:57.050143 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.049863 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-util\") pod \"15a7254a-c81f-416d-aeb3-38635e345123\" (UID: \"15a7254a-c81f-416d-aeb3-38635e345123\") " Jul 30 10:55:57.050436 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.050409 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-bundle" (OuterVolumeSpecName: "bundle") pod "15a7254a-c81f-416d-aeb3-38635e345123" (UID: "15a7254a-c81f-416d-aeb3-38635e345123"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:55:57.050753 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.050537 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-bundle" (OuterVolumeSpecName: "bundle") pod "f4501bd0-504b-43d9-8034-770a20df60c2" (UID: "f4501bd0-504b-43d9-8034-770a20df60c2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:55:57.050904 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.050727 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-bundle" (OuterVolumeSpecName: "bundle") pod "cf443429-dcd0-4405-94cf-d2718d49b287" (UID: "cf443429-dcd0-4405-94cf-d2718d49b287"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:55:57.052345 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.052310 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4501bd0-504b-43d9-8034-770a20df60c2-kube-api-access-cddpc" (OuterVolumeSpecName: "kube-api-access-cddpc") pod "f4501bd0-504b-43d9-8034-770a20df60c2" (UID: "f4501bd0-504b-43d9-8034-770a20df60c2"). InnerVolumeSpecName "kube-api-access-cddpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:55:57.052452 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.052375 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a7254a-c81f-416d-aeb3-38635e345123-kube-api-access-sml2m" (OuterVolumeSpecName: "kube-api-access-sml2m") pod "15a7254a-c81f-416d-aeb3-38635e345123" (UID: "15a7254a-c81f-416d-aeb3-38635e345123"). InnerVolumeSpecName "kube-api-access-sml2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:55:57.052642 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.052619 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf443429-dcd0-4405-94cf-d2718d49b287-kube-api-access-m4jsk" (OuterVolumeSpecName: "kube-api-access-m4jsk") pod "cf443429-dcd0-4405-94cf-d2718d49b287" (UID: "cf443429-dcd0-4405-94cf-d2718d49b287"). InnerVolumeSpecName "kube-api-access-m4jsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:55:57.056631 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.056609 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-util" (OuterVolumeSpecName: "util") pod "f4501bd0-504b-43d9-8034-770a20df60c2" (UID: "f4501bd0-504b-43d9-8034-770a20df60c2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:55:57.056726 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.056670 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-util" (OuterVolumeSpecName: "util") pod "15a7254a-c81f-416d-aeb3-38635e345123" (UID: "15a7254a-c81f-416d-aeb3-38635e345123"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:55:57.057464 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.057440 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-util" (OuterVolumeSpecName: "util") pod "cf443429-dcd0-4405-94cf-d2718d49b287" (UID: "cf443429-dcd0-4405-94cf-d2718d49b287"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:55:57.150730 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.150690 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-bundle\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:57.150730 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.150725 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cddpc\" (UniqueName: \"kubernetes.io/projected/f4501bd0-504b-43d9-8034-770a20df60c2-kube-api-access-cddpc\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:57.150730 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.150736 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-util\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:57.150969 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.150745 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-bundle\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:57.150969 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.150753 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4501bd0-504b-43d9-8034-770a20df60c2-bundle\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:57.150969 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.150765 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4jsk\" (UniqueName: \"kubernetes.io/projected/cf443429-dcd0-4405-94cf-d2718d49b287-kube-api-access-m4jsk\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:57.150969 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.150776 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf443429-dcd0-4405-94cf-d2718d49b287-util\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:57.150969 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.150784 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15a7254a-c81f-416d-aeb3-38635e345123-util\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:57.150969 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.150791 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sml2m\" (UniqueName: \"kubernetes.io/projected/15a7254a-c81f-416d-aeb3-38635e345123-kube-api-access-sml2m\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:57.773248 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.773214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" event={"ID":"f4501bd0-504b-43d9-8034-770a20df60c2","Type":"ContainerDied","Data":"2ba5d0c9a425b7a45cb67f862891760fb8d16bd5fed54da6693a1c51cd37ac7e"} Jul 30 10:55:57.773700 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.773265 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ba5d0c9a425b7a45cb67f862891760fb8d16bd5fed54da6693a1c51cd37ac7e" Jul 30 10:55:57.773700 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.773224 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c479fz" Jul 30 10:55:57.775080 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.775063 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" Jul 30 10:55:57.775216 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.775113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bbljmvf" event={"ID":"15a7254a-c81f-416d-aeb3-38635e345123","Type":"ContainerDied","Data":"af81f4f49268095fa8c3effe1f7aedfdfbffdba2a10272a91f4d8d92110f6ac4"} Jul 30 10:55:57.775216 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.775154 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af81f4f49268095fa8c3effe1f7aedfdfbffdba2a10272a91f4d8d92110f6ac4" Jul 30 10:55:57.777308 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.777281 2575 generic.go:356] "Generic (PLEG): container finished" podID="e118f86a-cac1-43fd-9e92-84d465cba441" containerID="89029eeccb3833c14b4a40f0f0b75e1a6ed2d9193e00cff8f0ce9c678ce870b2" exitCode=0 Jul 30 10:55:57.777402 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.777364 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" event={"ID":"e118f86a-cac1-43fd-9e92-84d465cba441","Type":"ContainerDied","Data":"89029eeccb3833c14b4a40f0f0b75e1a6ed2d9193e00cff8f0ce9c678ce870b2"} Jul 30 10:55:57.779233 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.779214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" event={"ID":"cf443429-dcd0-4405-94cf-d2718d49b287","Type":"ContainerDied","Data":"f03d4daa7ba13fb6d3bc96ec60000013660cd323f9ea654cbae280149d003e57"} Jul 30 10:55:57.779326 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.779241 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870534wcq2" Jul 30 10:55:57.779326 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:57.779247 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f03d4daa7ba13fb6d3bc96ec60000013660cd323f9ea654cbae280149d003e57" Jul 30 10:55:58.919277 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:58.919251 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:58.968661 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:58.968614 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-bundle\") pod \"e118f86a-cac1-43fd-9e92-84d465cba441\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " Jul 30 10:55:58.968862 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:58.968699 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-util\") pod \"e118f86a-cac1-43fd-9e92-84d465cba441\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " Jul 30 10:55:58.968862 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:58.968745 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn68r\" (UniqueName: \"kubernetes.io/projected/e118f86a-cac1-43fd-9e92-84d465cba441-kube-api-access-hn68r\") pod \"e118f86a-cac1-43fd-9e92-84d465cba441\" (UID: \"e118f86a-cac1-43fd-9e92-84d465cba441\") " Jul 30 10:55:58.969221 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:58.969180 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-bundle" (OuterVolumeSpecName: "bundle") pod "e118f86a-cac1-43fd-9e92-84d465cba441" (UID: "e118f86a-cac1-43fd-9e92-84d465cba441"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:55:58.970873 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:58.970843 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e118f86a-cac1-43fd-9e92-84d465cba441-kube-api-access-hn68r" (OuterVolumeSpecName: "kube-api-access-hn68r") pod "e118f86a-cac1-43fd-9e92-84d465cba441" (UID: "e118f86a-cac1-43fd-9e92-84d465cba441"). InnerVolumeSpecName "kube-api-access-hn68r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:55:58.973816 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:58.973794 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-util" (OuterVolumeSpecName: "util") pod "e118f86a-cac1-43fd-9e92-84d465cba441" (UID: "e118f86a-cac1-43fd-9e92-84d465cba441"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:55:59.070149 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:59.070016 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-util\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:59.070149 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:59.070060 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn68r\" (UniqueName: \"kubernetes.io/projected/e118f86a-cac1-43fd-9e92-84d465cba441-kube-api-access-hn68r\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:59.070149 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:59.070070 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e118f86a-cac1-43fd-9e92-84d465cba441-bundle\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:55:59.792441 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:59.792416 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" Jul 30 10:55:59.792613 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:59.792409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0hpmb7" event={"ID":"e118f86a-cac1-43fd-9e92-84d465cba441","Type":"ContainerDied","Data":"da2117faeeeaf264524e7b7a85faad140ba4fd55df4f03ea3649aa7fdc46272a"} Jul 30 10:55:59.792613 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:55:59.792533 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da2117faeeeaf264524e7b7a85faad140ba4fd55df4f03ea3649aa7fdc46272a" Jul 30 10:56:10.892060 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892031 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx"] Jul 30 10:56:10.892473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892404 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a7254a-c81f-416d-aeb3-38635e345123" containerName="pull" Jul 30 10:56:10.892473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892416 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a7254a-c81f-416d-aeb3-38635e345123" containerName="pull" Jul 30 10:56:10.892473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892425 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4501bd0-504b-43d9-8034-770a20df60c2" containerName="extract" Jul 30 10:56:10.892473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892432 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4501bd0-504b-43d9-8034-770a20df60c2" containerName="extract" Jul 30 10:56:10.892473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892441 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e118f86a-cac1-43fd-9e92-84d465cba441" containerName="extract" Jul 30 10:56:10.892473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892446 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e118f86a-cac1-43fd-9e92-84d465cba441" containerName="extract" Jul 30 10:56:10.892473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892457 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf443429-dcd0-4405-94cf-d2718d49b287" containerName="pull" Jul 30 10:56:10.892473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892462 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf443429-dcd0-4405-94cf-d2718d49b287" containerName="pull" Jul 30 10:56:10.892473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892471 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf443429-dcd0-4405-94cf-d2718d49b287" containerName="util" Jul 30 10:56:10.892473 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892478 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf443429-dcd0-4405-94cf-d2718d49b287" containerName="util" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892486 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4501bd0-504b-43d9-8034-770a20df60c2" containerName="util" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892493 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4501bd0-504b-43d9-8034-770a20df60c2" containerName="util" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892498 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e118f86a-cac1-43fd-9e92-84d465cba441" containerName="pull" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892503 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e118f86a-cac1-43fd-9e92-84d465cba441" containerName="pull" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892512 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e118f86a-cac1-43fd-9e92-84d465cba441" containerName="util" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892517 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e118f86a-cac1-43fd-9e92-84d465cba441" containerName="util" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892522 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a7254a-c81f-416d-aeb3-38635e345123" containerName="util" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892527 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a7254a-c81f-416d-aeb3-38635e345123" containerName="util" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892533 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4501bd0-504b-43d9-8034-770a20df60c2" containerName="pull" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892540 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4501bd0-504b-43d9-8034-770a20df60c2" containerName="pull" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892548 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf443429-dcd0-4405-94cf-d2718d49b287" containerName="extract" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892553 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf443429-dcd0-4405-94cf-d2718d49b287" containerName="extract" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892558 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15a7254a-c81f-416d-aeb3-38635e345123" containerName="extract" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892563 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="15a7254a-c81f-416d-aeb3-38635e345123" containerName="extract" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892612 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf443429-dcd0-4405-94cf-d2718d49b287" containerName="extract" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892620 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4501bd0-504b-43d9-8034-770a20df60c2" containerName="extract" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892626 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="15a7254a-c81f-416d-aeb3-38635e345123" containerName="extract" Jul 30 10:56:10.892799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.892633 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e118f86a-cac1-43fd-9e92-84d465cba441" containerName="extract" Jul 30 10:56:10.895545 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.895530 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx" Jul 30 10:56:10.897506 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.897479 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 30 10:56:10.897622 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.897518 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-m9nvv\"" Jul 30 10:56:10.897622 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.897520 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 30 10:56:10.897848 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.897834 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 30 10:56:10.904402 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.904381 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx"] Jul 30 10:56:10.976218 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:10.976180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhmm\" (UniqueName: \"kubernetes.io/projected/bcbcd504-d10a-4db5-b254-05f7e39ed612-kube-api-access-llhmm\") pod \"dns-operator-controller-manager-5985564776-rk8vx\" (UID: \"bcbcd504-d10a-4db5-b254-05f7e39ed612\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx" Jul 30 10:56:11.077482 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:11.077448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhmm\" (UniqueName: \"kubernetes.io/projected/bcbcd504-d10a-4db5-b254-05f7e39ed612-kube-api-access-llhmm\") pod \"dns-operator-controller-manager-5985564776-rk8vx\" (UID: \"bcbcd504-d10a-4db5-b254-05f7e39ed612\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx" Jul 30 10:56:11.085021 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:11.084985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhmm\" (UniqueName: \"kubernetes.io/projected/bcbcd504-d10a-4db5-b254-05f7e39ed612-kube-api-access-llhmm\") pod \"dns-operator-controller-manager-5985564776-rk8vx\" (UID: \"bcbcd504-d10a-4db5-b254-05f7e39ed612\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx" Jul 30 10:56:11.206387 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:11.206293 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx" Jul 30 10:56:11.367683 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:11.367656 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx"] Jul 30 10:56:11.369355 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:56:11.369329 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbcd504_d10a_4db5_b254_05f7e39ed612.slice/crio-25d171c2c557424413ca84cbe0d34a3d7782528da6ce9119e69943a15f5db5d1 WatchSource:0}: Error finding container 25d171c2c557424413ca84cbe0d34a3d7782528da6ce9119e69943a15f5db5d1: Status 404 returned error can't find the container with id 25d171c2c557424413ca84cbe0d34a3d7782528da6ce9119e69943a15f5db5d1 Jul 30 10:56:11.858149 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:11.858113 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx" event={"ID":"bcbcd504-d10a-4db5-b254-05f7e39ed612","Type":"ContainerStarted","Data":"25d171c2c557424413ca84cbe0d34a3d7782528da6ce9119e69943a15f5db5d1"} Jul 30 10:56:13.827427 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:13.827402 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 30 10:56:14.875704 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:14.875669 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx" event={"ID":"bcbcd504-d10a-4db5-b254-05f7e39ed612","Type":"ContainerStarted","Data":"7f9dc4790dec102ac68d254cb792564a6c7bffc8b164cce19a30b0a5c86e214a"} Jul 30 10:56:14.876130 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:14.875814 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx" Jul 30 10:56:14.891130 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:14.891052 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx" podStartSLOduration=2.437519487 podStartE2EDuration="4.891037221s" totalImagesPullingTime="2.453517734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:56:14.890764896 +0000 UTC m=+482.863625930" watchObservedRunningTime="2026-07-30 10:56:14.891037221 +0000 UTC m=+482.863898262" Jul 30 10:56:15.226809 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.226727 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9"] Jul 30 10:56:15.230279 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.230258 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:15.233526 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.233505 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fqxbm\"" Jul 30 10:56:15.249454 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.249417 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9"] Jul 30 10:56:15.317066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.317026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6134afdc-6516-479d-8e9e-adc5f6ac55d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" (UID: \"6134afdc-6516-479d-8e9e-adc5f6ac55d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:15.317271 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.317083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf5zq\" (UniqueName: \"kubernetes.io/projected/6134afdc-6516-479d-8e9e-adc5f6ac55d0-kube-api-access-pf5zq\") pod \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" (UID: \"6134afdc-6516-479d-8e9e-adc5f6ac55d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:15.418382 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.418340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6134afdc-6516-479d-8e9e-adc5f6ac55d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" (UID: \"6134afdc-6516-479d-8e9e-adc5f6ac55d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:15.418572 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.418414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf5zq\" (UniqueName: \"kubernetes.io/projected/6134afdc-6516-479d-8e9e-adc5f6ac55d0-kube-api-access-pf5zq\") pod \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" (UID: \"6134afdc-6516-479d-8e9e-adc5f6ac55d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:15.418806 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.418784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6134afdc-6516-479d-8e9e-adc5f6ac55d0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" (UID: \"6134afdc-6516-479d-8e9e-adc5f6ac55d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:15.435256 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.435221 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf5zq\" (UniqueName: \"kubernetes.io/projected/6134afdc-6516-479d-8e9e-adc5f6ac55d0-kube-api-access-pf5zq\") pod \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" (UID: \"6134afdc-6516-479d-8e9e-adc5f6ac55d0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:15.541724 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.541690 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:15.682822 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.682794 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9"] Jul 30 10:56:15.684929 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:56:15.684903 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6134afdc_6516_479d_8e9e_adc5f6ac55d0.slice/crio-663aaaba78216c6eb59ee7cd3f63e3e108b6df07f362096298646c2c0105319c WatchSource:0}: Error finding container 663aaaba78216c6eb59ee7cd3f63e3e108b6df07f362096298646c2c0105319c: Status 404 returned error can't find the container with id 663aaaba78216c6eb59ee7cd3f63e3e108b6df07f362096298646c2c0105319c Jul 30 10:56:15.882119 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:15.882007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" event={"ID":"6134afdc-6516-479d-8e9e-adc5f6ac55d0","Type":"ContainerStarted","Data":"663aaaba78216c6eb59ee7cd3f63e3e108b6df07f362096298646c2c0105319c"} Jul 30 10:56:17.456790 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:17.456748 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8"] Jul 30 10:56:17.460572 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:17.460549 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:17.462772 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:17.462748 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-48h28\"" Jul 30 10:56:17.471405 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:17.471337 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8"] Jul 30 10:56:17.542871 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:17.542832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xsn8\" (UniqueName: \"kubernetes.io/projected/135aa85b-3fdd-426e-9632-ef774118f359-kube-api-access-5xsn8\") pod \"limitador-operator-controller-manager-6c549b6d85-twhs8\" (UID: \"135aa85b-3fdd-426e-9632-ef774118f359\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:17.643767 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:17.643726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xsn8\" (UniqueName: \"kubernetes.io/projected/135aa85b-3fdd-426e-9632-ef774118f359-kube-api-access-5xsn8\") pod \"limitador-operator-controller-manager-6c549b6d85-twhs8\" (UID: \"135aa85b-3fdd-426e-9632-ef774118f359\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:17.653349 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:17.653320 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xsn8\" (UniqueName: \"kubernetes.io/projected/135aa85b-3fdd-426e-9632-ef774118f359-kube-api-access-5xsn8\") pod \"limitador-operator-controller-manager-6c549b6d85-twhs8\" (UID: \"135aa85b-3fdd-426e-9632-ef774118f359\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:17.776077 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:17.775988 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:17.956290 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:17.956254 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8"] Jul 30 10:56:17.957476 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:56:17.957444 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135aa85b_3fdd_426e_9632_ef774118f359.slice/crio-ef1495df6b10deca289b8c234a07bea0175a58e51350950497043c9e1d400af3 WatchSource:0}: Error finding container ef1495df6b10deca289b8c234a07bea0175a58e51350950497043c9e1d400af3: Status 404 returned error can't find the container with id ef1495df6b10deca289b8c234a07bea0175a58e51350950497043c9e1d400af3 Jul 30 10:56:18.909547 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:18.909505 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" event={"ID":"135aa85b-3fdd-426e-9632-ef774118f359","Type":"ContainerStarted","Data":"ef1495df6b10deca289b8c234a07bea0175a58e51350950497043c9e1d400af3"} Jul 30 10:56:19.916971 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:19.916928 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" event={"ID":"6134afdc-6516-479d-8e9e-adc5f6ac55d0","Type":"ContainerStarted","Data":"11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c"} Jul 30 10:56:19.917392 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:19.916991 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:19.935821 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:19.935747 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" podStartSLOduration=1.223130505 podStartE2EDuration="4.935728765s" totalImagesPullingTime="3.71259826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:56:19.935185526 +0000 UTC m=+487.908046574" watchObservedRunningTime="2026-07-30 10:56:19.935728765 +0000 UTC m=+487.908589813" Jul 30 10:56:20.924387 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:20.924337 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" event={"ID":"135aa85b-3fdd-426e-9632-ef774118f359","Type":"ContainerStarted","Data":"ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5"} Jul 30 10:56:20.924763 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:20.924420 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:20.946182 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:20.946126 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" podStartSLOduration=1.096791819 podStartE2EDuration="3.946108797s" totalImagesPullingTime="2.849316978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:56:20.943938529 +0000 UTC m=+488.916799582" watchObservedRunningTime="2026-07-30 10:56:20.946108797 +0000 UTC m=+488.918969828" Jul 30 10:56:25.884735 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:25.884699 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-rk8vx" Jul 30 10:56:30.927443 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:30.927404 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:31.932418 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:31.932385 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:32.642813 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.642779 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9"] Jul 30 10:56:32.643016 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.642961 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" containerName="manager" containerID="cri-o://11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c" gracePeriod=2 Jul 30 10:56:32.648696 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.648662 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9"] Jul 30 10:56:32.675154 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.672151 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8"] Jul 30 10:56:32.675154 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.672509 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" podUID="135aa85b-3fdd-426e-9632-ef774118f359" containerName="manager" containerID="cri-o://ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5" gracePeriod=2 Jul 30 10:56:32.680025 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.679988 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8"] Jul 30 10:56:32.684374 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.684324 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:32.685919 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.685884 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:32.687547 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.687522 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf"] Jul 30 10:56:32.687991 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.687969 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135aa85b-3fdd-426e-9632-ef774118f359" containerName="manager" Jul 30 10:56:32.687991 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.687994 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="135aa85b-3fdd-426e-9632-ef774118f359" containerName="manager" Jul 30 10:56:32.688181 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.688016 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" containerName="manager" Jul 30 10:56:32.688181 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.688023 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" containerName="manager" Jul 30 10:56:32.688181 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.688137 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" containerName="manager" Jul 30 10:56:32.688181 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.688170 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="135aa85b-3fdd-426e-9632-ef774118f359" containerName="manager" Jul 30 10:56:32.691554 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.691530 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss"] Jul 30 10:56:32.691886 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.691863 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:32.693987 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.693951 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:32.695461 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.695417 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:32.695571 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.695462 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss" Jul 30 10:56:32.696915 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.696890 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf"] Jul 30 10:56:32.707174 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.707139 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss"] Jul 30 10:56:32.720642 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.720600 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:32.722955 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.722732 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:32.785481 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.785414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-h2fqf\" (UID: \"be993f5c-7efc-4aca-ae71-01f8efcc3d3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:32.785568 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.785486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqp5r\" (UniqueName: \"kubernetes.io/projected/45a7ed45-e7c6-4faa-b60b-1f2c62584111-kube-api-access-nqp5r\") pod \"limitador-operator-controller-manager-6c549b6d85-zqmss\" (UID: \"45a7ed45-e7c6-4faa-b60b-1f2c62584111\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss" Jul 30 10:56:32.785631 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.785604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7kf\" (UniqueName: \"kubernetes.io/projected/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-kube-api-access-dt7kf\") pod \"kuadrant-operator-controller-manager-58fc56757c-h2fqf\" (UID: \"be993f5c-7efc-4aca-ae71-01f8efcc3d3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:32.886474 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.886442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7kf\" (UniqueName: \"kubernetes.io/projected/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-kube-api-access-dt7kf\") pod \"kuadrant-operator-controller-manager-58fc56757c-h2fqf\" (UID: \"be993f5c-7efc-4aca-ae71-01f8efcc3d3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:32.886658 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.886491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-h2fqf\" (UID: \"be993f5c-7efc-4aca-ae71-01f8efcc3d3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:32.886658 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.886543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqp5r\" (UniqueName: \"kubernetes.io/projected/45a7ed45-e7c6-4faa-b60b-1f2c62584111-kube-api-access-nqp5r\") pod \"limitador-operator-controller-manager-6c549b6d85-zqmss\" (UID: \"45a7ed45-e7c6-4faa-b60b-1f2c62584111\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss" Jul 30 10:56:32.886957 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.886934 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-h2fqf\" (UID: \"be993f5c-7efc-4aca-ae71-01f8efcc3d3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:32.896142 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.896049 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqp5r\" (UniqueName: \"kubernetes.io/projected/45a7ed45-e7c6-4faa-b60b-1f2c62584111-kube-api-access-nqp5r\") pod \"limitador-operator-controller-manager-6c549b6d85-zqmss\" (UID: \"45a7ed45-e7c6-4faa-b60b-1f2c62584111\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss" Jul 30 10:56:32.896255 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.896235 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7kf\" (UniqueName: \"kubernetes.io/projected/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-kube-api-access-dt7kf\") pod \"kuadrant-operator-controller-manager-58fc56757c-h2fqf\" (UID: \"be993f5c-7efc-4aca-ae71-01f8efcc3d3c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:32.930107 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.930063 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:32.931808 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.931781 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:32.932177 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.932162 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:32.933182 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.933157 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:32.934490 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.934468 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:32.935705 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.935687 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:32.987560 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.987527 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf5zq\" (UniqueName: \"kubernetes.io/projected/6134afdc-6516-479d-8e9e-adc5f6ac55d0-kube-api-access-pf5zq\") pod \"6134afdc-6516-479d-8e9e-adc5f6ac55d0\" (UID: \"6134afdc-6516-479d-8e9e-adc5f6ac55d0\") " Jul 30 10:56:32.987751 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.987576 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6134afdc-6516-479d-8e9e-adc5f6ac55d0-extensions-socket-volume\") pod \"6134afdc-6516-479d-8e9e-adc5f6ac55d0\" (UID: \"6134afdc-6516-479d-8e9e-adc5f6ac55d0\") " Jul 30 10:56:32.987751 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.987596 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xsn8\" (UniqueName: \"kubernetes.io/projected/135aa85b-3fdd-426e-9632-ef774118f359-kube-api-access-5xsn8\") pod \"135aa85b-3fdd-426e-9632-ef774118f359\" (UID: \"135aa85b-3fdd-426e-9632-ef774118f359\") " Jul 30 10:56:32.988165 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.988135 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6134afdc-6516-479d-8e9e-adc5f6ac55d0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6134afdc-6516-479d-8e9e-adc5f6ac55d0" (UID: "6134afdc-6516-479d-8e9e-adc5f6ac55d0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:56:32.989893 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.989866 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6134afdc-6516-479d-8e9e-adc5f6ac55d0-kube-api-access-pf5zq" (OuterVolumeSpecName: "kube-api-access-pf5zq") pod "6134afdc-6516-479d-8e9e-adc5f6ac55d0" (UID: "6134afdc-6516-479d-8e9e-adc5f6ac55d0"). InnerVolumeSpecName "kube-api-access-pf5zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:56:32.989991 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.989905 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135aa85b-3fdd-426e-9632-ef774118f359-kube-api-access-5xsn8" (OuterVolumeSpecName: "kube-api-access-5xsn8") pod "135aa85b-3fdd-426e-9632-ef774118f359" (UID: "135aa85b-3fdd-426e-9632-ef774118f359"). InnerVolumeSpecName "kube-api-access-5xsn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:56:32.992591 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.992563 2575 generic.go:356] "Generic (PLEG): container finished" podID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" containerID="11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c" exitCode=0 Jul 30 10:56:32.992680 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.992608 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:32.992680 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.992659 2575 scope.go:117] "RemoveContainer" containerID="11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c" Jul 30 10:56:32.994195 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.994173 2575 generic.go:356] "Generic (PLEG): container finished" podID="135aa85b-3fdd-426e-9632-ef774118f359" containerID="ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5" exitCode=0 Jul 30 10:56:32.994279 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:32.994221 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:32.994506 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.994481 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:32.995820 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.995785 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:32.997194 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.997167 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:32.998484 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:32.998452 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:33.003603 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.003580 2575 scope.go:117] "RemoveContainer" containerID="11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c" Jul 30 10:56:33.003952 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:33.003882 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c\": container with ID starting with 11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c not found: ID does not exist" containerID="11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c" Jul 30 10:56:33.003952 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.003906 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c"} err="failed to get container status \"11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c\": rpc error: code = NotFound desc = could not find container \"11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c\": container with ID starting with 11d3d445b84b19d93b5752096610ba8e202ea25c7160af1e4bd625535ad1cf6c not found: ID does not exist" Jul 30 10:56:33.003952 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.003938 2575 scope.go:117] "RemoveContainer" containerID="ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5" Jul 30 10:56:33.004249 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:33.004016 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:33.005521 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:33.005496 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:33.007104 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:33.007069 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:33.008376 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:33.008335 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:33.012623 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.012602 2575 scope.go:117] "RemoveContainer" containerID="ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5" Jul 30 10:56:33.012909 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:33.012890 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5\": container with ID starting with ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5 not found: ID does not exist" containerID="ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5" Jul 30 10:56:33.012972 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.012917 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5"} err="failed to get container status \"ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5\": rpc error: code = NotFound desc = could not find container \"ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5\": container with ID starting with ca083b314592425644142aea92e9fe37413eeaebdb0701626714c216fb5960f5 not found: ID does not exist" Jul 30 10:56:33.088466 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.088426 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:33.088654 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.088630 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf5zq\" (UniqueName: \"kubernetes.io/projected/6134afdc-6516-479d-8e9e-adc5f6ac55d0-kube-api-access-pf5zq\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:56:33.088714 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.088660 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6134afdc-6516-479d-8e9e-adc5f6ac55d0-extensions-socket-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:56:33.088714 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.088669 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xsn8\" (UniqueName: \"kubernetes.io/projected/135aa85b-3fdd-426e-9632-ef774118f359-kube-api-access-5xsn8\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:56:33.094835 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.094798 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss" Jul 30 10:56:33.244540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.244513 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf"] Jul 30 10:56:33.247467 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:56:33.247438 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe993f5c_7efc_4aca_ae71_01f8efcc3d3c.slice/crio-b032b9a9d30739de332908fa9a43ad1c9390884e9e6ec29de1d265bddb8aeae0 WatchSource:0}: Error finding container b032b9a9d30739de332908fa9a43ad1c9390884e9e6ec29de1d265bddb8aeae0: Status 404 returned error can't find the container with id b032b9a9d30739de332908fa9a43ad1c9390884e9e6ec29de1d265bddb8aeae0 Jul 30 10:56:33.270929 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.270900 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss"] Jul 30 10:56:33.272780 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:56:33.272750 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a7ed45_e7c6_4faa_b60b_1f2c62584111.slice/crio-4a8ebe13cb416bab231ebf76676f92117642f1442b67dfe1d3f952c1b54b099f WatchSource:0}: Error finding container 4a8ebe13cb416bab231ebf76676f92117642f1442b67dfe1d3f952c1b54b099f: Status 404 returned error can't find the container with id 4a8ebe13cb416bab231ebf76676f92117642f1442b67dfe1d3f952c1b54b099f Jul 30 10:56:33.561808 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.561767 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw"] Jul 30 10:56:33.565523 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.565499 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 10:56:33.567242 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:33.567207 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:33.586204 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.586155 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw"] Jul 30 10:56:33.594535 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.594501 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dq2n\" (UniqueName: \"kubernetes.io/projected/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-kube-api-access-4dq2n\") pod \"kuadrant-operator-controller-manager-68cf589589-v6czw\" (UID: \"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 10:56:33.594736 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.594556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-v6czw\" (UID: \"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 10:56:33.597435 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:33.597384 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:33.695811 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.695772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dq2n\" (UniqueName: \"kubernetes.io/projected/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-kube-api-access-4dq2n\") pod \"kuadrant-operator-controller-manager-68cf589589-v6czw\" (UID: \"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 10:56:33.695990 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.695826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-v6czw\" (UID: \"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 10:56:33.696276 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.696258 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-v6czw\" (UID: \"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 10:56:33.703768 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.703740 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dq2n\" (UniqueName: \"kubernetes.io/projected/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-kube-api-access-4dq2n\") pod \"kuadrant-operator-controller-manager-68cf589589-v6czw\" (UID: \"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 10:56:33.875799 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:33.875695 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 10:56:34.002050 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.002009 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss" event={"ID":"45a7ed45-e7c6-4faa-b60b-1f2c62584111","Type":"ContainerStarted","Data":"d635ab06b3cd9b6c78c23048476070328a8b768d339ece9d1b08ac011aa440bf"} Jul 30 10:56:34.002511 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.002068 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss" event={"ID":"45a7ed45-e7c6-4faa-b60b-1f2c62584111","Type":"ContainerStarted","Data":"4a8ebe13cb416bab231ebf76676f92117642f1442b67dfe1d3f952c1b54b099f"} Jul 30 10:56:34.002511 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.002119 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss" Jul 30 10:56:34.004000 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:34.003972 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:34.005756 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.005732 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" event={"ID":"be993f5c-7efc-4aca-ae71-01f8efcc3d3c","Type":"ContainerStarted","Data":"fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce"} Jul 30 10:56:34.005851 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.005779 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" event={"ID":"be993f5c-7efc-4aca-ae71-01f8efcc3d3c","Type":"ContainerStarted","Data":"b032b9a9d30739de332908fa9a43ad1c9390884e9e6ec29de1d265bddb8aeae0"} Jul 30 10:56:34.005851 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.005803 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:34.022644 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.022587 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss" podStartSLOduration=2.022567811 podStartE2EDuration="2.022567811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:56:34.020783262 +0000 UTC m=+501.993644297" watchObservedRunningTime="2026-07-30 10:56:34.022567811 +0000 UTC m=+501.995428846" Jul 30 10:56:34.022797 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:34.022613 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:34.029130 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.028954 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw"] Jul 30 10:56:34.033291 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:56:34.033246 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3057b1fb_9c43_4ef9_90e5_e82086ebd8c0.slice/crio-a6e25172fc0e8eaa5e8ab90663dda904a77f33fb85e04e6125e048bd3d29c828 WatchSource:0}: Error finding container a6e25172fc0e8eaa5e8ab90663dda904a77f33fb85e04e6125e048bd3d29c828: Status 404 returned error can't find the container with id a6e25172fc0e8eaa5e8ab90663dda904a77f33fb85e04e6125e048bd3d29c828 Jul 30 10:56:34.045313 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:34.045280 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-twhs8\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="135aa85b-3fdd-426e-9632-ef774118f359" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-twhs8" Jul 30 10:56:34.046401 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.046354 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" podStartSLOduration=2.046335551 podStartE2EDuration="2.046335551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:56:34.043680427 +0000 UTC m=+502.016541494" watchObservedRunningTime="2026-07-30 10:56:34.046335551 +0000 UTC m=+502.019196587" Jul 30 10:56:34.046477 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:34.046459 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-czpd9\" is forbidden: User \"system:node:ip-10-0-138-131.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-131.ec2.internal' and this object" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-czpd9" Jul 30 10:56:34.532109 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.528895 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="135aa85b-3fdd-426e-9632-ef774118f359" path="/var/lib/kubelet/pods/135aa85b-3fdd-426e-9632-ef774118f359/volumes" Jul 30 10:56:34.532109 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:34.529550 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6134afdc-6516-479d-8e9e-adc5f6ac55d0" path="/var/lib/kubelet/pods/6134afdc-6516-479d-8e9e-adc5f6ac55d0/volumes" Jul 30 10:56:35.015315 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:35.015209 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" event={"ID":"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0","Type":"ContainerStarted","Data":"0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b"} Jul 30 10:56:35.015751 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:35.015439 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 10:56:35.015751 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:35.015483 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" event={"ID":"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0","Type":"ContainerStarted","Data":"a6e25172fc0e8eaa5e8ab90663dda904a77f33fb85e04e6125e048bd3d29c828"} Jul 30 10:56:35.038011 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:35.037951 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" podStartSLOduration=2.037931093 podStartE2EDuration="2.037931093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:56:35.034940736 +0000 UTC m=+503.007801804" watchObservedRunningTime="2026-07-30 10:56:35.037931093 +0000 UTC m=+503.010792128" Jul 30 10:56:45.017777 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:45.017744 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-zqmss" Jul 30 10:56:45.018273 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:45.017916 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:46.024631 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:46.024591 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 10:56:46.077438 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:46.077403 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf"] Jul 30 10:56:46.077705 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:46.077654 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" podUID="be993f5c-7efc-4aca-ae71-01f8efcc3d3c" containerName="manager" containerID="cri-o://fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce" gracePeriod=10 Jul 30 10:56:46.341445 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:46.341400 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:46.414536 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:46.414494 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-extensions-socket-volume\") pod \"be993f5c-7efc-4aca-ae71-01f8efcc3d3c\" (UID: \"be993f5c-7efc-4aca-ae71-01f8efcc3d3c\") " Jul 30 10:56:46.414721 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:46.414577 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt7kf\" (UniqueName: \"kubernetes.io/projected/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-kube-api-access-dt7kf\") pod \"be993f5c-7efc-4aca-ae71-01f8efcc3d3c\" (UID: \"be993f5c-7efc-4aca-ae71-01f8efcc3d3c\") " Jul 30 10:56:46.414997 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:46.414966 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "be993f5c-7efc-4aca-ae71-01f8efcc3d3c" (UID: "be993f5c-7efc-4aca-ae71-01f8efcc3d3c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 10:56:46.416676 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:46.416651 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-kube-api-access-dt7kf" (OuterVolumeSpecName: "kube-api-access-dt7kf") pod "be993f5c-7efc-4aca-ae71-01f8efcc3d3c" (UID: "be993f5c-7efc-4aca-ae71-01f8efcc3d3c"). InnerVolumeSpecName "kube-api-access-dt7kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:56:46.516017 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:46.515973 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-extensions-socket-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:56:46.516017 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:46.516009 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt7kf\" (UniqueName: \"kubernetes.io/projected/be993f5c-7efc-4aca-ae71-01f8efcc3d3c-kube-api-access-dt7kf\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:56:47.086573 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:47.086537 2575 generic.go:356] "Generic (PLEG): container finished" podID="be993f5c-7efc-4aca-ae71-01f8efcc3d3c" containerID="fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce" exitCode=0 Jul 30 10:56:47.087002 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:47.086584 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" Jul 30 10:56:47.087002 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:47.086585 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" event={"ID":"be993f5c-7efc-4aca-ae71-01f8efcc3d3c","Type":"ContainerDied","Data":"fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce"} Jul 30 10:56:47.087002 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:47.086708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf" event={"ID":"be993f5c-7efc-4aca-ae71-01f8efcc3d3c","Type":"ContainerDied","Data":"b032b9a9d30739de332908fa9a43ad1c9390884e9e6ec29de1d265bddb8aeae0"} Jul 30 10:56:47.087002 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:47.086738 2575 scope.go:117] "RemoveContainer" containerID="fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce" Jul 30 10:56:47.095819 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:47.095795 2575 scope.go:117] "RemoveContainer" containerID="fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce" Jul 30 10:56:47.096155 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:56:47.096135 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce\": container with ID starting with fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce not found: ID does not exist" containerID="fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce" Jul 30 10:56:47.096226 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:47.096163 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce"} err="failed to get container status \"fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce\": rpc error: code = NotFound desc = could not find container \"fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce\": container with ID starting with fc14a98aa6ce28dede81263db2fe1ea6856e05e2ed9625bdd581f906fa82ecce not found: ID does not exist" Jul 30 10:56:47.109826 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:47.109789 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf"] Jul 30 10:56:47.115003 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:47.114974 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-h2fqf"] Jul 30 10:56:48.527959 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:56:48.527925 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be993f5c-7efc-4aca-ae71-01f8efcc3d3c" path="/var/lib/kubelet/pods/be993f5c-7efc-4aca-ae71-01f8efcc3d3c/volumes" Jul 30 10:57:05.065127 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.065022 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-jstnx"] Jul 30 10:57:05.065518 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.065468 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be993f5c-7efc-4aca-ae71-01f8efcc3d3c" containerName="manager" Jul 30 10:57:05.065518 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.065480 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="be993f5c-7efc-4aca-ae71-01f8efcc3d3c" containerName="manager" Jul 30 10:57:05.065609 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.065567 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="be993f5c-7efc-4aca-ae71-01f8efcc3d3c" containerName="manager" Jul 30 10:57:05.067782 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.067760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:05.069561 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.069532 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 30 10:57:05.069703 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.069581 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qjxbw\"" Jul 30 10:57:05.079371 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.079336 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-jstnx"] Jul 30 10:57:05.175712 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.175679 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-jstnx"] Jul 30 10:57:05.187573 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.187542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-config-file\") pod \"limitador-limitador-859c4d96d6-jstnx\" (UID: \"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:05.187755 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.187608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xpw\" (UniqueName: \"kubernetes.io/projected/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-kube-api-access-w2xpw\") pod \"limitador-limitador-859c4d96d6-jstnx\" (UID: \"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:05.289051 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.289012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-config-file\") pod \"limitador-limitador-859c4d96d6-jstnx\" (UID: \"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:05.289306 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.289070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xpw\" (UniqueName: \"kubernetes.io/projected/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-kube-api-access-w2xpw\") pod \"limitador-limitador-859c4d96d6-jstnx\" (UID: \"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:05.289743 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.289714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-config-file\") pod \"limitador-limitador-859c4d96d6-jstnx\" (UID: \"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:05.296697 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.296669 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xpw\" (UniqueName: \"kubernetes.io/projected/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-kube-api-access-w2xpw\") pod \"limitador-limitador-859c4d96d6-jstnx\" (UID: \"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:05.380306 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.380206 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:05.524329 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:05.524302 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-jstnx"] Jul 30 10:57:05.526835 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:57:05.526788 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a46b102_c35b_4f7a_953a_7a5c0ea5d09b.slice/crio-77627dd9241fc98d3a586fa9d7099ebcf2d5d220677b8a6410a9b0b5e4e1e666 WatchSource:0}: Error finding container 77627dd9241fc98d3a586fa9d7099ebcf2d5d220677b8a6410a9b0b5e4e1e666: Status 404 returned error can't find the container with id 77627dd9241fc98d3a586fa9d7099ebcf2d5d220677b8a6410a9b0b5e4e1e666 Jul 30 10:57:06.080605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:06.080570 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vx56j"] Jul 30 10:57:06.083695 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:06.083670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vx56j" Jul 30 10:57:06.085661 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:06.085639 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x9m5r\"" Jul 30 10:57:06.092340 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:06.091586 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vx56j"] Jul 30 10:57:06.197795 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:06.197765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57d6d\" (UniqueName: \"kubernetes.io/projected/2b288e6a-4ab1-40bd-8dc2-d11835774e70-kube-api-access-57d6d\") pod \"authorino-c548b5d7d-vx56j\" (UID: \"2b288e6a-4ab1-40bd-8dc2-d11835774e70\") " pod="kuadrant-system/authorino-c548b5d7d-vx56j" Jul 30 10:57:06.198773 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:06.198746 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" event={"ID":"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b","Type":"ContainerStarted","Data":"77627dd9241fc98d3a586fa9d7099ebcf2d5d220677b8a6410a9b0b5e4e1e666"} Jul 30 10:57:06.298958 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:06.298914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57d6d\" (UniqueName: \"kubernetes.io/projected/2b288e6a-4ab1-40bd-8dc2-d11835774e70-kube-api-access-57d6d\") pod \"authorino-c548b5d7d-vx56j\" (UID: \"2b288e6a-4ab1-40bd-8dc2-d11835774e70\") " pod="kuadrant-system/authorino-c548b5d7d-vx56j" Jul 30 10:57:06.307605 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:06.307570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57d6d\" (UniqueName: \"kubernetes.io/projected/2b288e6a-4ab1-40bd-8dc2-d11835774e70-kube-api-access-57d6d\") pod \"authorino-c548b5d7d-vx56j\" (UID: \"2b288e6a-4ab1-40bd-8dc2-d11835774e70\") " pod="kuadrant-system/authorino-c548b5d7d-vx56j" Jul 30 10:57:06.394885 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:06.394793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vx56j" Jul 30 10:57:06.531468 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:06.531439 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vx56j"] Jul 30 10:57:06.534013 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:57:06.533978 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b288e6a_4ab1_40bd_8dc2_d11835774e70.slice/crio-40cd6cc91fb3973188e3b4ad99d1725d7ef5a33f9dacd8fab295308c8d34e280 WatchSource:0}: Error finding container 40cd6cc91fb3973188e3b4ad99d1725d7ef5a33f9dacd8fab295308c8d34e280: Status 404 returned error can't find the container with id 40cd6cc91fb3973188e3b4ad99d1725d7ef5a33f9dacd8fab295308c8d34e280 Jul 30 10:57:07.206385 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:07.206282 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vx56j" event={"ID":"2b288e6a-4ab1-40bd-8dc2-d11835774e70","Type":"ContainerStarted","Data":"40cd6cc91fb3973188e3b4ad99d1725d7ef5a33f9dacd8fab295308c8d34e280"} Jul 30 10:57:07.207854 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:07.207829 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" event={"ID":"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b","Type":"ContainerStarted","Data":"412722c80a50ced4ccc7a9cbee4c8b79681a5af097eb19961a0f9d3bbc3b5cf9"} Jul 30 10:57:07.207984 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:07.207907 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:07.233041 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:07.232936 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" podStartSLOduration=0.819248198 podStartE2EDuration="2.232913005s" totalImagesPullingTime="1.413664807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:57:07.224212258 +0000 UTC m=+535.197073309" watchObservedRunningTime="2026-07-30 10:57:07.232913005 +0000 UTC m=+535.205774041" Jul 30 10:57:09.221737 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:09.221697 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vx56j" event={"ID":"2b288e6a-4ab1-40bd-8dc2-d11835774e70","Type":"ContainerStarted","Data":"bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf"} Jul 30 10:57:09.236759 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:09.236697 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-vx56j" podStartSLOduration=1.1245439 podStartE2EDuration="3.236680579s" totalImagesPullingTime="2.112136679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:57:09.235906273 +0000 UTC m=+537.208767310" watchObservedRunningTime="2026-07-30 10:57:09.236680579 +0000 UTC m=+537.209541613" Jul 30 10:57:18.215737 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:18.215708 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:19.758137 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:19.758076 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-jstnx"] Jul 30 10:57:19.758892 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:19.758830 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" podUID="5a46b102-c35b-4f7a-953a-7a5c0ea5d09b" containerName="limitador" containerID="cri-o://412722c80a50ced4ccc7a9cbee4c8b79681a5af097eb19961a0f9d3bbc3b5cf9" gracePeriod=30 Jul 30 10:57:20.288478 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:20.288444 2575 generic.go:356] "Generic (PLEG): container finished" podID="5a46b102-c35b-4f7a-953a-7a5c0ea5d09b" containerID="412722c80a50ced4ccc7a9cbee4c8b79681a5af097eb19961a0f9d3bbc3b5cf9" exitCode=0 Jul 30 10:57:20.288660 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:20.288516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" event={"ID":"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b","Type":"ContainerDied","Data":"412722c80a50ced4ccc7a9cbee4c8b79681a5af097eb19961a0f9d3bbc3b5cf9"} Jul 30 10:57:20.310814 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:20.310784 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:20.324573 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:20.324478 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-config-file\") pod \"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b\" (UID: \"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b\") " Jul 30 10:57:20.324573 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:20.324544 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2xpw\" (UniqueName: \"kubernetes.io/projected/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-kube-api-access-w2xpw\") pod \"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b\" (UID: \"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b\") " Jul 30 10:57:20.324881 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:20.324857 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-config-file" (OuterVolumeSpecName: "config-file") pod "5a46b102-c35b-4f7a-953a-7a5c0ea5d09b" (UID: "5a46b102-c35b-4f7a-953a-7a5c0ea5d09b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 10:57:20.327012 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:20.326980 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-kube-api-access-w2xpw" (OuterVolumeSpecName: "kube-api-access-w2xpw") pod "5a46b102-c35b-4f7a-953a-7a5c0ea5d09b" (UID: "5a46b102-c35b-4f7a-953a-7a5c0ea5d09b"). InnerVolumeSpecName "kube-api-access-w2xpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:57:20.425310 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:20.425276 2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-config-file\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:57:20.425310 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:20.425307 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xpw\" (UniqueName: \"kubernetes.io/projected/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b-kube-api-access-w2xpw\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:57:21.295270 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.295240 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" Jul 30 10:57:21.295749 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.295268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-jstnx" event={"ID":"5a46b102-c35b-4f7a-953a-7a5c0ea5d09b","Type":"ContainerDied","Data":"77627dd9241fc98d3a586fa9d7099ebcf2d5d220677b8a6410a9b0b5e4e1e666"} Jul 30 10:57:21.295749 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.295337 2575 scope.go:117] "RemoveContainer" containerID="412722c80a50ced4ccc7a9cbee4c8b79681a5af097eb19961a0f9d3bbc3b5cf9" Jul 30 10:57:21.310890 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.310861 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-jstnx"] Jul 30 10:57:21.314540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.314509 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-jstnx"] Jul 30 10:57:21.536365 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.536321 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w"] Jul 30 10:57:21.536749 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.536735 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a46b102-c35b-4f7a-953a-7a5c0ea5d09b" containerName="limitador" Jul 30 10:57:21.536793 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.536750 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a46b102-c35b-4f7a-953a-7a5c0ea5d09b" containerName="limitador" Jul 30 10:57:21.536833 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.536811 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a46b102-c35b-4f7a-953a-7a5c0ea5d09b" containerName="limitador" Jul 30 10:57:21.539723 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.539699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" Jul 30 10:57:21.541965 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.541812 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 30 10:57:21.541965 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.541840 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6zt7m\"" Jul 30 10:57:21.541965 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.541851 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 30 10:57:21.541965 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.541875 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 30 10:57:21.550555 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.550529 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w"] Jul 30 10:57:21.635381 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.635337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2g5q\" (UniqueName: \"kubernetes.io/projected/fe27bc51-0b91-4df6-b459-cdb6e93b4873-kube-api-access-v2g5q\") pod \"postgres-69fb4594cf-jtf5w\" (UID: \"fe27bc51-0b91-4df6-b459-cdb6e93b4873\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" Jul 30 10:57:21.635574 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.635456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fe27bc51-0b91-4df6-b459-cdb6e93b4873-data\") pod \"postgres-69fb4594cf-jtf5w\" (UID: \"fe27bc51-0b91-4df6-b459-cdb6e93b4873\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" Jul 30 10:57:21.736416 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.736376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2g5q\" (UniqueName: \"kubernetes.io/projected/fe27bc51-0b91-4df6-b459-cdb6e93b4873-kube-api-access-v2g5q\") pod \"postgres-69fb4594cf-jtf5w\" (UID: \"fe27bc51-0b91-4df6-b459-cdb6e93b4873\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" Jul 30 10:57:21.736644 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.736453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fe27bc51-0b91-4df6-b459-cdb6e93b4873-data\") pod \"postgres-69fb4594cf-jtf5w\" (UID: \"fe27bc51-0b91-4df6-b459-cdb6e93b4873\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" Jul 30 10:57:21.736956 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.736936 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fe27bc51-0b91-4df6-b459-cdb6e93b4873-data\") pod \"postgres-69fb4594cf-jtf5w\" (UID: \"fe27bc51-0b91-4df6-b459-cdb6e93b4873\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" Jul 30 10:57:21.743957 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.743935 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2g5q\" (UniqueName: \"kubernetes.io/projected/fe27bc51-0b91-4df6-b459-cdb6e93b4873-kube-api-access-v2g5q\") pod \"postgres-69fb4594cf-jtf5w\" (UID: \"fe27bc51-0b91-4df6-b459-cdb6e93b4873\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" Jul 30 10:57:21.853624 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.853510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" Jul 30 10:57:21.998960 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:21.998930 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w"] Jul 30 10:57:22.001054 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:57:22.001025 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe27bc51_0b91_4df6_b459_cdb6e93b4873.slice/crio-f2b4a86e2069fd6155364e9f45ec6bd936be977ef34f99948fe24e927aa9fac4 WatchSource:0}: Error finding container f2b4a86e2069fd6155364e9f45ec6bd936be977ef34f99948fe24e927aa9fac4: Status 404 returned error can't find the container with id f2b4a86e2069fd6155364e9f45ec6bd936be977ef34f99948fe24e927aa9fac4 Jul 30 10:57:22.302807 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:22.302771 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" event={"ID":"fe27bc51-0b91-4df6-b459-cdb6e93b4873","Type":"ContainerStarted","Data":"f2b4a86e2069fd6155364e9f45ec6bd936be977ef34f99948fe24e927aa9fac4"} Jul 30 10:57:22.527564 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:22.527523 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a46b102-c35b-4f7a-953a-7a5c0ea5d09b" path="/var/lib/kubelet/pods/5a46b102-c35b-4f7a-953a-7a5c0ea5d09b/volumes" Jul 30 10:57:27.336389 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:27.336345 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" event={"ID":"fe27bc51-0b91-4df6-b459-cdb6e93b4873","Type":"ContainerStarted","Data":"acab3eb1d9d4056959e16526645fbcf05f708983a3195eaa8eb113195ff4fbaa"} Jul 30 10:57:27.336807 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:27.336427 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" Jul 30 10:57:27.352212 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:27.352165 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" podStartSLOduration=1.3951268510000001 podStartE2EDuration="6.352149773s" totalImagesPullingTime="4.957022922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:57:27.350761632 +0000 UTC m=+555.323622668" watchObservedRunningTime="2026-07-30 10:57:27.352149773 +0000 UTC m=+555.325010807" Jul 30 10:57:33.371793 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:33.371763 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jtf5w" Jul 30 10:57:33.907648 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:33.907608 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xgkzk"] Jul 30 10:57:33.910796 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:33.910772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xgkzk" Jul 30 10:57:33.918678 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:33.918377 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xgkzk"] Jul 30 10:57:33.946459 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:33.946422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wf8\" (UniqueName: \"kubernetes.io/projected/b9c3f2b6-c993-48c0-8aec-6095d0c42a21-kube-api-access-r6wf8\") pod \"authorino-5dc7ccb5f-xgkzk\" (UID: \"b9c3f2b6-c993-48c0-8aec-6095d0c42a21\") " pod="kuadrant-system/authorino-5dc7ccb5f-xgkzk" Jul 30 10:57:33.997345 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:33.997312 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xgkzk"] Jul 30 10:57:33.997631 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:57:33.997612 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-r6wf8], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-xgkzk" podUID="b9c3f2b6-c993-48c0-8aec-6095d0c42a21" Jul 30 10:57:34.025736 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.025700 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-7sd72"] Jul 30 10:57:34.027991 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.027974 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-7sd72" Jul 30 10:57:34.030164 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.030142 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 30 10:57:34.037998 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.037975 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-7sd72"] Jul 30 10:57:34.047915 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.047882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wf8\" (UniqueName: \"kubernetes.io/projected/b9c3f2b6-c993-48c0-8aec-6095d0c42a21-kube-api-access-r6wf8\") pod \"authorino-5dc7ccb5f-xgkzk\" (UID: \"b9c3f2b6-c993-48c0-8aec-6095d0c42a21\") " pod="kuadrant-system/authorino-5dc7ccb5f-xgkzk" Jul 30 10:57:34.056279 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.056244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wf8\" (UniqueName: \"kubernetes.io/projected/b9c3f2b6-c993-48c0-8aec-6095d0c42a21-kube-api-access-r6wf8\") pod \"authorino-5dc7ccb5f-xgkzk\" (UID: \"b9c3f2b6-c993-48c0-8aec-6095d0c42a21\") " pod="kuadrant-system/authorino-5dc7ccb5f-xgkzk" Jul 30 10:57:34.149457 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.149419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/510ddfad-13d0-4d08-bd21-4eb3e0721222-tls-cert\") pod \"authorino-687bd45585-7sd72\" (UID: \"510ddfad-13d0-4d08-bd21-4eb3e0721222\") " pod="kuadrant-system/authorino-687bd45585-7sd72" Jul 30 10:57:34.149643 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.149472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52gfl\" (UniqueName: \"kubernetes.io/projected/510ddfad-13d0-4d08-bd21-4eb3e0721222-kube-api-access-52gfl\") pod \"authorino-687bd45585-7sd72\" (UID: \"510ddfad-13d0-4d08-bd21-4eb3e0721222\") " pod="kuadrant-system/authorino-687bd45585-7sd72" Jul 30 10:57:34.159048 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.158958 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-7sd72"] Jul 30 10:57:34.159284 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:57:34.159263 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-52gfl tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-7sd72" podUID="510ddfad-13d0-4d08-bd21-4eb3e0721222" Jul 30 10:57:34.250833 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.250799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/510ddfad-13d0-4d08-bd21-4eb3e0721222-tls-cert\") pod \"authorino-687bd45585-7sd72\" (UID: \"510ddfad-13d0-4d08-bd21-4eb3e0721222\") " pod="kuadrant-system/authorino-687bd45585-7sd72" Jul 30 10:57:34.251021 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.250845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gfl\" (UniqueName: \"kubernetes.io/projected/510ddfad-13d0-4d08-bd21-4eb3e0721222-kube-api-access-52gfl\") pod \"authorino-687bd45585-7sd72\" (UID: \"510ddfad-13d0-4d08-bd21-4eb3e0721222\") " pod="kuadrant-system/authorino-687bd45585-7sd72" Jul 30 10:57:34.253371 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.253338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/510ddfad-13d0-4d08-bd21-4eb3e0721222-tls-cert\") pod \"authorino-687bd45585-7sd72\" (UID: \"510ddfad-13d0-4d08-bd21-4eb3e0721222\") " pod="kuadrant-system/authorino-687bd45585-7sd72" Jul 30 10:57:34.258746 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.258716 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gfl\" (UniqueName: \"kubernetes.io/projected/510ddfad-13d0-4d08-bd21-4eb3e0721222-kube-api-access-52gfl\") pod \"authorino-687bd45585-7sd72\" (UID: \"510ddfad-13d0-4d08-bd21-4eb3e0721222\") " pod="kuadrant-system/authorino-687bd45585-7sd72" Jul 30 10:57:34.375402 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.375377 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xgkzk" Jul 30 10:57:34.375797 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.375377 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-7sd72" Jul 30 10:57:34.382119 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.382077 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xgkzk" Jul 30 10:57:34.385608 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.385586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-7sd72" Jul 30 10:57:34.452705 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.452619 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6wf8\" (UniqueName: \"kubernetes.io/projected/b9c3f2b6-c993-48c0-8aec-6095d0c42a21-kube-api-access-r6wf8\") pod \"b9c3f2b6-c993-48c0-8aec-6095d0c42a21\" (UID: \"b9c3f2b6-c993-48c0-8aec-6095d0c42a21\") " Jul 30 10:57:34.452705 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.452659 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/510ddfad-13d0-4d08-bd21-4eb3e0721222-tls-cert\") pod \"510ddfad-13d0-4d08-bd21-4eb3e0721222\" (UID: \"510ddfad-13d0-4d08-bd21-4eb3e0721222\") " Jul 30 10:57:34.452705 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.452685 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52gfl\" (UniqueName: \"kubernetes.io/projected/510ddfad-13d0-4d08-bd21-4eb3e0721222-kube-api-access-52gfl\") pod \"510ddfad-13d0-4d08-bd21-4eb3e0721222\" (UID: \"510ddfad-13d0-4d08-bd21-4eb3e0721222\") " Jul 30 10:57:34.454853 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.454817 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510ddfad-13d0-4d08-bd21-4eb3e0721222-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "510ddfad-13d0-4d08-bd21-4eb3e0721222" (UID: "510ddfad-13d0-4d08-bd21-4eb3e0721222"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:57:34.454853 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.454833 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510ddfad-13d0-4d08-bd21-4eb3e0721222-kube-api-access-52gfl" (OuterVolumeSpecName: "kube-api-access-52gfl") pod "510ddfad-13d0-4d08-bd21-4eb3e0721222" (UID: "510ddfad-13d0-4d08-bd21-4eb3e0721222"). InnerVolumeSpecName "kube-api-access-52gfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:57:34.454853 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.454823 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c3f2b6-c993-48c0-8aec-6095d0c42a21-kube-api-access-r6wf8" (OuterVolumeSpecName: "kube-api-access-r6wf8") pod "b9c3f2b6-c993-48c0-8aec-6095d0c42a21" (UID: "b9c3f2b6-c993-48c0-8aec-6095d0c42a21"). InnerVolumeSpecName "kube-api-access-r6wf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:57:34.554330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.554282 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6wf8\" (UniqueName: \"kubernetes.io/projected/b9c3f2b6-c993-48c0-8aec-6095d0c42a21-kube-api-access-r6wf8\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:57:34.554330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.554321 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/510ddfad-13d0-4d08-bd21-4eb3e0721222-tls-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:57:34.554330 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:34.554337 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52gfl\" (UniqueName: \"kubernetes.io/projected/510ddfad-13d0-4d08-bd21-4eb3e0721222-kube-api-access-52gfl\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:57:35.381071 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:35.381036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xgkzk" Jul 30 10:57:35.381492 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:35.381036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-7sd72" Jul 30 10:57:35.406914 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:35.406877 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xgkzk"] Jul 30 10:57:35.410296 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:35.410264 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xgkzk"] Jul 30 10:57:35.437073 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:35.437038 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-7sd72"] Jul 30 10:57:35.442120 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:35.442060 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-7sd72"] Jul 30 10:57:35.856528 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:35.856495 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vx56j"] Jul 30 10:57:35.856759 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:35.856703 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-vx56j" podUID="2b288e6a-4ab1-40bd-8dc2-d11835774e70" containerName="authorino" containerID="cri-o://bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf" gracePeriod=30 Jul 30 10:57:36.125819 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.125793 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vx56j" Jul 30 10:57:36.168614 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.168570 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57d6d\" (UniqueName: \"kubernetes.io/projected/2b288e6a-4ab1-40bd-8dc2-d11835774e70-kube-api-access-57d6d\") pod \"2b288e6a-4ab1-40bd-8dc2-d11835774e70\" (UID: \"2b288e6a-4ab1-40bd-8dc2-d11835774e70\") " Jul 30 10:57:36.170654 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.170619 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b288e6a-4ab1-40bd-8dc2-d11835774e70-kube-api-access-57d6d" (OuterVolumeSpecName: "kube-api-access-57d6d") pod "2b288e6a-4ab1-40bd-8dc2-d11835774e70" (UID: "2b288e6a-4ab1-40bd-8dc2-d11835774e70"). InnerVolumeSpecName "kube-api-access-57d6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:57:36.269891 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.269847 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57d6d\" (UniqueName: \"kubernetes.io/projected/2b288e6a-4ab1-40bd-8dc2-d11835774e70-kube-api-access-57d6d\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:57:36.388500 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.388411 2575 generic.go:356] "Generic (PLEG): container finished" podID="2b288e6a-4ab1-40bd-8dc2-d11835774e70" containerID="bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf" exitCode=0 Jul 30 10:57:36.388500 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.388457 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-vx56j" Jul 30 10:57:36.388500 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.388493 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vx56j" event={"ID":"2b288e6a-4ab1-40bd-8dc2-d11835774e70","Type":"ContainerDied","Data":"bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf"} Jul 30 10:57:36.389032 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.388532 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-vx56j" event={"ID":"2b288e6a-4ab1-40bd-8dc2-d11835774e70","Type":"ContainerDied","Data":"40cd6cc91fb3973188e3b4ad99d1725d7ef5a33f9dacd8fab295308c8d34e280"} Jul 30 10:57:36.389032 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.388550 2575 scope.go:117] "RemoveContainer" containerID="bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf" Jul 30 10:57:36.398662 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.398642 2575 scope.go:117] "RemoveContainer" containerID="bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf" Jul 30 10:57:36.398958 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:57:36.398939 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf\": container with ID starting with bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf not found: ID does not exist" containerID="bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf" Jul 30 10:57:36.399016 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.398968 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf"} err="failed to get container status \"bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf\": rpc error: code = NotFound desc = could not find container \"bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf\": container with ID starting with bc2320d99241caa848202b00135e73614868ac106bcc52aced94a75309944edf not found: ID does not exist" Jul 30 10:57:36.410420 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.410384 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vx56j"] Jul 30 10:57:36.413906 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.413875 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-vx56j"] Jul 30 10:57:36.528065 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.528030 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b288e6a-4ab1-40bd-8dc2-d11835774e70" path="/var/lib/kubelet/pods/2b288e6a-4ab1-40bd-8dc2-d11835774e70/volumes" Jul 30 10:57:36.528446 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.528432 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="510ddfad-13d0-4d08-bd21-4eb3e0721222" path="/var/lib/kubelet/pods/510ddfad-13d0-4d08-bd21-4eb3e0721222/volumes" Jul 30 10:57:36.528631 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:36.528620 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9c3f2b6-c993-48c0-8aec-6095d0c42a21" path="/var/lib/kubelet/pods/b9c3f2b6-c993-48c0-8aec-6095d0c42a21/volumes" Jul 30 10:57:40.585907 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.585872 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-77bccc884d-wzvl5"] Jul 30 10:57:40.586397 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.586359 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b288e6a-4ab1-40bd-8dc2-d11835774e70" containerName="authorino" Jul 30 10:57:40.586397 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.586372 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b288e6a-4ab1-40bd-8dc2-d11835774e70" containerName="authorino" Jul 30 10:57:40.586470 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.586447 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b288e6a-4ab1-40bd-8dc2-d11835774e70" containerName="authorino" Jul 30 10:57:40.749058 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.749021 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77bccc884d-wzvl5"] Jul 30 10:57:40.749058 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.749047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:40.751110 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.751074 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 10:57:40.751427 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.751412 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 10:57:40.751427 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.751424 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ghk2x\"" Jul 30 10:57:40.767042 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.767014 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d558b5f66-2fkmh"] Jul 30 10:57:40.790772 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.790746 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-2fkmh"] Jul 30 10:57:40.790925 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.790841 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 10:57:40.813977 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.813944 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e98f4793-cc9b-4569-a4b5-86411d210e1e-webhook-cert\") pod \"maas-controller-77bccc884d-wzvl5\" (UID: \"e98f4793-cc9b-4569-a4b5-86411d210e1e\") " pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:40.814160 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.814007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94wv\" (UniqueName: \"kubernetes.io/projected/e98f4793-cc9b-4569-a4b5-86411d210e1e-kube-api-access-q94wv\") pod \"maas-controller-77bccc884d-wzvl5\" (UID: \"e98f4793-cc9b-4569-a4b5-86411d210e1e\") " pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:40.915387 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.915293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94wv\" (UniqueName: \"kubernetes.io/projected/e98f4793-cc9b-4569-a4b5-86411d210e1e-kube-api-access-q94wv\") pod \"maas-controller-77bccc884d-wzvl5\" (UID: \"e98f4793-cc9b-4569-a4b5-86411d210e1e\") " pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:40.915387 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.915350 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4479c\" (UniqueName: \"kubernetes.io/projected/f734ed02-8f10-4332-8d79-f77521c55ccc-kube-api-access-4479c\") pod \"maas-controller-6d558b5f66-2fkmh\" (UID: \"f734ed02-8f10-4332-8d79-f77521c55ccc\") " pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 10:57:40.915387 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.915388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f734ed02-8f10-4332-8d79-f77521c55ccc-webhook-cert\") pod \"maas-controller-6d558b5f66-2fkmh\" (UID: \"f734ed02-8f10-4332-8d79-f77521c55ccc\") " pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 10:57:40.915677 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.915453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e98f4793-cc9b-4569-a4b5-86411d210e1e-webhook-cert\") pod \"maas-controller-77bccc884d-wzvl5\" (UID: \"e98f4793-cc9b-4569-a4b5-86411d210e1e\") " pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:40.917815 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.917786 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e98f4793-cc9b-4569-a4b5-86411d210e1e-webhook-cert\") pod \"maas-controller-77bccc884d-wzvl5\" (UID: \"e98f4793-cc9b-4569-a4b5-86411d210e1e\") " pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:40.922840 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:40.922819 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94wv\" (UniqueName: \"kubernetes.io/projected/e98f4793-cc9b-4569-a4b5-86411d210e1e-kube-api-access-q94wv\") pod \"maas-controller-77bccc884d-wzvl5\" (UID: \"e98f4793-cc9b-4569-a4b5-86411d210e1e\") " pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:41.016685 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:41.016650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f734ed02-8f10-4332-8d79-f77521c55ccc-webhook-cert\") pod \"maas-controller-6d558b5f66-2fkmh\" (UID: \"f734ed02-8f10-4332-8d79-f77521c55ccc\") " pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 10:57:41.016866 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:41.016768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4479c\" (UniqueName: \"kubernetes.io/projected/f734ed02-8f10-4332-8d79-f77521c55ccc-kube-api-access-4479c\") pod \"maas-controller-6d558b5f66-2fkmh\" (UID: \"f734ed02-8f10-4332-8d79-f77521c55ccc\") " pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 10:57:41.019226 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:41.019193 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f734ed02-8f10-4332-8d79-f77521c55ccc-webhook-cert\") pod \"maas-controller-6d558b5f66-2fkmh\" (UID: \"f734ed02-8f10-4332-8d79-f77521c55ccc\") " pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 10:57:41.024946 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:41.024920 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4479c\" (UniqueName: \"kubernetes.io/projected/f734ed02-8f10-4332-8d79-f77521c55ccc-kube-api-access-4479c\") pod \"maas-controller-6d558b5f66-2fkmh\" (UID: \"f734ed02-8f10-4332-8d79-f77521c55ccc\") " pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 10:57:41.059834 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:41.059793 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:41.101274 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:41.101244 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 10:57:41.227601 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:41.227568 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-77bccc884d-wzvl5"] Jul 30 10:57:41.229548 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:57:41.229519 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98f4793_cc9b_4569_a4b5_86411d210e1e.slice/crio-ac22fc4908f41399374e9ac28059216902d823ee00cfbdc0e6f06aa631638b4f WatchSource:0}: Error finding container ac22fc4908f41399374e9ac28059216902d823ee00cfbdc0e6f06aa631638b4f: Status 404 returned error can't find the container with id ac22fc4908f41399374e9ac28059216902d823ee00cfbdc0e6f06aa631638b4f Jul 30 10:57:41.271050 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:41.271020 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-2fkmh"] Jul 30 10:57:41.272252 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:57:41.272228 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf734ed02_8f10_4332_8d79_f77521c55ccc.slice/crio-7f04b57cbdc6199bcc9ff4cee97e893b170b375d206fae0e276b76240e63de1e WatchSource:0}: Error finding container 7f04b57cbdc6199bcc9ff4cee97e893b170b375d206fae0e276b76240e63de1e: Status 404 returned error can't find the container with id 7f04b57cbdc6199bcc9ff4cee97e893b170b375d206fae0e276b76240e63de1e Jul 30 10:57:41.422222 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:41.422123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" event={"ID":"f734ed02-8f10-4332-8d79-f77521c55ccc","Type":"ContainerStarted","Data":"7f04b57cbdc6199bcc9ff4cee97e893b170b375d206fae0e276b76240e63de1e"} Jul 30 10:57:41.423462 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:41.423432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bccc884d-wzvl5" event={"ID":"e98f4793-cc9b-4569-a4b5-86411d210e1e","Type":"ContainerStarted","Data":"ac22fc4908f41399374e9ac28059216902d823ee00cfbdc0e6f06aa631638b4f"} Jul 30 10:57:44.446835 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:44.446802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" event={"ID":"f734ed02-8f10-4332-8d79-f77521c55ccc","Type":"ContainerStarted","Data":"bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd"} Jul 30 10:57:44.447339 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:44.446877 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 10:57:44.448462 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:44.448432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bccc884d-wzvl5" event={"ID":"e98f4793-cc9b-4569-a4b5-86411d210e1e","Type":"ContainerStarted","Data":"0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558"} Jul 30 10:57:44.448577 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:44.448561 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:44.466924 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:44.466869 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" podStartSLOduration=2.036952722 podStartE2EDuration="4.466849609s" totalImagesPullingTime="2.429896887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:57:44.466256285 +0000 UTC m=+572.439117332" watchObservedRunningTime="2026-07-30 10:57:44.466849609 +0000 UTC m=+572.439710708" Jul 30 10:57:44.484948 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:44.484891 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-77bccc884d-wzvl5" podStartSLOduration=2.018380043 podStartE2EDuration="4.484874019s" totalImagesPullingTime="2.466493976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:57:44.482820557 +0000 UTC m=+572.455681613" watchObservedRunningTime="2026-07-30 10:57:44.484874019 +0000 UTC m=+572.457735089" Jul 30 10:57:55.457080 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:55.457045 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:55.457502 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:55.457211 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 10:57:55.523111 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:55.523060 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77bccc884d-wzvl5"] Jul 30 10:57:55.523310 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:55.523287 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-77bccc884d-wzvl5" podUID="e98f4793-cc9b-4569-a4b5-86411d210e1e" containerName="manager" containerID="cri-o://0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558" gracePeriod=30 Jul 30 10:57:55.772266 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:55.772243 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:55.960393 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:55.960341 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e98f4793-cc9b-4569-a4b5-86411d210e1e-webhook-cert\") pod \"e98f4793-cc9b-4569-a4b5-86411d210e1e\" (UID: \"e98f4793-cc9b-4569-a4b5-86411d210e1e\") " Jul 30 10:57:55.960393 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:55.960398 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q94wv\" (UniqueName: \"kubernetes.io/projected/e98f4793-cc9b-4569-a4b5-86411d210e1e-kube-api-access-q94wv\") pod \"e98f4793-cc9b-4569-a4b5-86411d210e1e\" (UID: \"e98f4793-cc9b-4569-a4b5-86411d210e1e\") " Jul 30 10:57:55.962584 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:55.962548 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e98f4793-cc9b-4569-a4b5-86411d210e1e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e98f4793-cc9b-4569-a4b5-86411d210e1e" (UID: "e98f4793-cc9b-4569-a4b5-86411d210e1e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 10:57:55.962703 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:55.962626 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98f4793-cc9b-4569-a4b5-86411d210e1e-kube-api-access-q94wv" (OuterVolumeSpecName: "kube-api-access-q94wv") pod "e98f4793-cc9b-4569-a4b5-86411d210e1e" (UID: "e98f4793-cc9b-4569-a4b5-86411d210e1e"). InnerVolumeSpecName "kube-api-access-q94wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 10:57:56.061244 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.061198 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e98f4793-cc9b-4569-a4b5-86411d210e1e-webhook-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:57:56.061244 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.061242 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q94wv\" (UniqueName: \"kubernetes.io/projected/e98f4793-cc9b-4569-a4b5-86411d210e1e-kube-api-access-q94wv\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 10:57:56.524767 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.524677 2575 generic.go:356] "Generic (PLEG): container finished" podID="e98f4793-cc9b-4569-a4b5-86411d210e1e" containerID="0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558" exitCode=0 Jul 30 10:57:56.525268 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.524794 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-77bccc884d-wzvl5" Jul 30 10:57:56.528082 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.528057 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bccc884d-wzvl5" event={"ID":"e98f4793-cc9b-4569-a4b5-86411d210e1e","Type":"ContainerDied","Data":"0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558"} Jul 30 10:57:56.528239 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.528153 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-77bccc884d-wzvl5" event={"ID":"e98f4793-cc9b-4569-a4b5-86411d210e1e","Type":"ContainerDied","Data":"ac22fc4908f41399374e9ac28059216902d823ee00cfbdc0e6f06aa631638b4f"} Jul 30 10:57:56.528239 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.528179 2575 scope.go:117] "RemoveContainer" containerID="0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558" Jul 30 10:57:56.538480 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.538459 2575 scope.go:117] "RemoveContainer" containerID="0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558" Jul 30 10:57:56.538783 ip-10-0-138-131 kubenswrapper[2575]: E0730 10:57:56.538763 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558\": container with ID starting with 0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558 not found: ID does not exist" containerID="0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558" Jul 30 10:57:56.538858 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.538796 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558"} err="failed to get container status \"0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558\": rpc error: code = NotFound desc = could not find container \"0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558\": container with ID starting with 0e4ae8b9d67f3afeb15d4388ff7b4506860644d1c3c7892d9a21ece1b8ead558 not found: ID does not exist" Jul 30 10:57:56.548534 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.548503 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-77bccc884d-wzvl5"] Jul 30 10:57:56.552066 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:56.552043 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-77bccc884d-wzvl5"] Jul 30 10:57:58.526942 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:57:58.526910 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e98f4793-cc9b-4569-a4b5-86411d210e1e" path="/var/lib/kubelet/pods/e98f4793-cc9b-4569-a4b5-86411d210e1e/volumes" Jul 30 10:58:12.452237 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:12.452210 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 10:58:12.455345 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:12.455325 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 10:58:17.347930 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.347884 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-669686cd4-rhjj8"] Jul 30 10:58:17.348369 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.348359 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e98f4793-cc9b-4569-a4b5-86411d210e1e" containerName="manager" Jul 30 10:58:17.348422 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.348372 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e98f4793-cc9b-4569-a4b5-86411d210e1e" containerName="manager" Jul 30 10:58:17.348466 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.348455 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e98f4793-cc9b-4569-a4b5-86411d210e1e" containerName="manager" Jul 30 10:58:17.352933 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.352915 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.355427 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.355399 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gvjcm\"" Jul 30 10:58:17.355427 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.355422 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 30 10:58:17.362823 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.362799 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-669686cd4-rhjj8"] Jul 30 10:58:17.446897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.446859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69wg\" (UniqueName: \"kubernetes.io/projected/d2f549d3-8110-42c6-ae80-d46a2c102a62-kube-api-access-v69wg\") pod \"maas-api-669686cd4-rhjj8\" (UID: \"d2f549d3-8110-42c6-ae80-d46a2c102a62\") " pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.447118 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.446946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f549d3-8110-42c6-ae80-d46a2c102a62-maas-api-tls\") pod \"maas-api-669686cd4-rhjj8\" (UID: \"d2f549d3-8110-42c6-ae80-d46a2c102a62\") " pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.447118 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.446975 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f549d3-8110-42c6-ae80-d46a2c102a62-openshift-service-ca\") pod \"maas-api-669686cd4-rhjj8\" (UID: \"d2f549d3-8110-42c6-ae80-d46a2c102a62\") " pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.547572 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.547531 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69wg\" (UniqueName: \"kubernetes.io/projected/d2f549d3-8110-42c6-ae80-d46a2c102a62-kube-api-access-v69wg\") pod \"maas-api-669686cd4-rhjj8\" (UID: \"d2f549d3-8110-42c6-ae80-d46a2c102a62\") " pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.547786 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.547581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f549d3-8110-42c6-ae80-d46a2c102a62-maas-api-tls\") pod \"maas-api-669686cd4-rhjj8\" (UID: \"d2f549d3-8110-42c6-ae80-d46a2c102a62\") " pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.547786 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.547605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f549d3-8110-42c6-ae80-d46a2c102a62-openshift-service-ca\") pod \"maas-api-669686cd4-rhjj8\" (UID: \"d2f549d3-8110-42c6-ae80-d46a2c102a62\") " pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.548415 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.548391 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f549d3-8110-42c6-ae80-d46a2c102a62-openshift-service-ca\") pod \"maas-api-669686cd4-rhjj8\" (UID: \"d2f549d3-8110-42c6-ae80-d46a2c102a62\") " pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.550151 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.550129 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2f549d3-8110-42c6-ae80-d46a2c102a62-maas-api-tls\") pod \"maas-api-669686cd4-rhjj8\" (UID: \"d2f549d3-8110-42c6-ae80-d46a2c102a62\") " pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.576949 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.576912 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69wg\" (UniqueName: \"kubernetes.io/projected/d2f549d3-8110-42c6-ae80-d46a2c102a62-kube-api-access-v69wg\") pod \"maas-api-669686cd4-rhjj8\" (UID: \"d2f549d3-8110-42c6-ae80-d46a2c102a62\") " pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.664016 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.663926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:17.825986 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.825952 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-669686cd4-rhjj8"] Jul 30 10:58:17.828452 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:58:17.828423 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f549d3_8110_42c6_ae80_d46a2c102a62.slice/crio-758b55ad649da41f62b39840ea3172443bb3044bcb4880f5b8ce0d33ac54965b WatchSource:0}: Error finding container 758b55ad649da41f62b39840ea3172443bb3044bcb4880f5b8ce0d33ac54965b: Status 404 returned error can't find the container with id 758b55ad649da41f62b39840ea3172443bb3044bcb4880f5b8ce0d33ac54965b Jul 30 10:58:17.921399 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.921313 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d"] Jul 30 10:58:17.927901 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.927870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" Jul 30 10:58:17.930098 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.930061 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 30 10:58:17.930348 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.930332 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-dtvzg\"" Jul 30 10:58:17.936623 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.936591 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d"] Jul 30 10:58:17.952002 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.951958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5t8b\" (UniqueName: \"kubernetes.io/projected/2d7944ed-a849-430c-b6eb-bb8bd57754c0-kube-api-access-p5t8b\") pod \"payload-pre-processing-687dbb6cf6-h8q2d\" (UID: \"2d7944ed-a849-430c-b6eb-bb8bd57754c0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" Jul 30 10:58:17.952397 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.952334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7944ed-a849-430c-b6eb-bb8bd57754c0-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-h8q2d\" (UID: \"2d7944ed-a849-430c-b6eb-bb8bd57754c0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" Jul 30 10:58:17.954692 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.954669 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-w2xrv"] Jul 30 10:58:17.958742 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.958725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" Jul 30 10:58:17.970334 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:17.970303 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-w2xrv"] Jul 30 10:58:18.054228 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.054163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5t8b\" (UniqueName: \"kubernetes.io/projected/2d7944ed-a849-430c-b6eb-bb8bd57754c0-kube-api-access-p5t8b\") pod \"payload-pre-processing-687dbb6cf6-h8q2d\" (UID: \"2d7944ed-a849-430c-b6eb-bb8bd57754c0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" Jul 30 10:58:18.054423 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.054270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7944ed-a849-430c-b6eb-bb8bd57754c0-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-h8q2d\" (UID: \"2d7944ed-a849-430c-b6eb-bb8bd57754c0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" Jul 30 10:58:18.054423 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.054301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baba2139-d918-4f6f-9eab-3d14183d9659-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-w2xrv\" (UID: \"baba2139-d918-4f6f-9eab-3d14183d9659\") " pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" Jul 30 10:58:18.054558 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.054419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzzgz\" (UniqueName: \"kubernetes.io/projected/baba2139-d918-4f6f-9eab-3d14183d9659-kube-api-access-hzzgz\") pod \"payload-processing-5cc6cd444b-w2xrv\" (UID: \"baba2139-d918-4f6f-9eab-3d14183d9659\") " pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" Jul 30 10:58:18.055354 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.055321 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d7944ed-a849-430c-b6eb-bb8bd57754c0-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-h8q2d\" (UID: \"2d7944ed-a849-430c-b6eb-bb8bd57754c0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" Jul 30 10:58:18.064272 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.064240 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5t8b\" (UniqueName: \"kubernetes.io/projected/2d7944ed-a849-430c-b6eb-bb8bd57754c0-kube-api-access-p5t8b\") pod \"payload-pre-processing-687dbb6cf6-h8q2d\" (UID: \"2d7944ed-a849-430c-b6eb-bb8bd57754c0\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" Jul 30 10:58:18.155511 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.155474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baba2139-d918-4f6f-9eab-3d14183d9659-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-w2xrv\" (UID: \"baba2139-d918-4f6f-9eab-3d14183d9659\") " pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" Jul 30 10:58:18.155699 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.155533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzzgz\" (UniqueName: \"kubernetes.io/projected/baba2139-d918-4f6f-9eab-3d14183d9659-kube-api-access-hzzgz\") pod \"payload-processing-5cc6cd444b-w2xrv\" (UID: \"baba2139-d918-4f6f-9eab-3d14183d9659\") " pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" Jul 30 10:58:18.156461 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.156433 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baba2139-d918-4f6f-9eab-3d14183d9659-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-w2xrv\" (UID: \"baba2139-d918-4f6f-9eab-3d14183d9659\") " pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" Jul 30 10:58:18.164526 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.164499 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzzgz\" (UniqueName: \"kubernetes.io/projected/baba2139-d918-4f6f-9eab-3d14183d9659-kube-api-access-hzzgz\") pod \"payload-processing-5cc6cd444b-w2xrv\" (UID: \"baba2139-d918-4f6f-9eab-3d14183d9659\") " pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" Jul 30 10:58:18.243405 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.243302 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" Jul 30 10:58:18.272885 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.272856 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" Jul 30 10:58:18.408570 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.408530 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d"] Jul 30 10:58:18.410876 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:58:18.410807 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7944ed_a849_430c_b6eb_bb8bd57754c0.slice/crio-27ff2f969d69a5e48b9bf44f21fc8a81474e20ff677ac3a024fc77c9eab02a75 WatchSource:0}: Error finding container 27ff2f969d69a5e48b9bf44f21fc8a81474e20ff677ac3a024fc77c9eab02a75: Status 404 returned error can't find the container with id 27ff2f969d69a5e48b9bf44f21fc8a81474e20ff677ac3a024fc77c9eab02a75 Jul 30 10:58:18.453894 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.453868 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-w2xrv"] Jul 30 10:58:18.456739 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:58:18.456704 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaba2139_d918_4f6f_9eab_3d14183d9659.slice/crio-39e1bbf01a3d2222320abecded13ce9436895af41d80175e0cdcdc2e76bcd869 WatchSource:0}: Error finding container 39e1bbf01a3d2222320abecded13ce9436895af41d80175e0cdcdc2e76bcd869: Status 404 returned error can't find the container with id 39e1bbf01a3d2222320abecded13ce9436895af41d80175e0cdcdc2e76bcd869 Jul 30 10:58:18.651992 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.651947 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" event={"ID":"baba2139-d918-4f6f-9eab-3d14183d9659","Type":"ContainerStarted","Data":"39e1bbf01a3d2222320abecded13ce9436895af41d80175e0cdcdc2e76bcd869"} Jul 30 10:58:18.653736 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.653704 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" event={"ID":"2d7944ed-a849-430c-b6eb-bb8bd57754c0","Type":"ContainerStarted","Data":"27ff2f969d69a5e48b9bf44f21fc8a81474e20ff677ac3a024fc77c9eab02a75"} Jul 30 10:58:18.655285 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:18.655253 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" event={"ID":"d2f549d3-8110-42c6-ae80-d46a2c102a62","Type":"ContainerStarted","Data":"758b55ad649da41f62b39840ea3172443bb3044bcb4880f5b8ce0d33ac54965b"} Jul 30 10:58:19.670416 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:19.670365 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" event={"ID":"d2f549d3-8110-42c6-ae80-d46a2c102a62","Type":"ContainerStarted","Data":"d2827db12f94e9f1d8075ba08cb9b102d42e1bb71ee2b941214802736a704edb"} Jul 30 10:58:19.670879 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:19.670491 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:19.691267 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:19.689441 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" podStartSLOduration=1.336545002 podStartE2EDuration="2.68941562s" totalImagesPullingTime="1.352870618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:58:19.688334683 +0000 UTC m=+607.661195720" watchObservedRunningTime="2026-07-30 10:58:19.68941562 +0000 UTC m=+607.662276657" Jul 30 10:58:20.681006 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:20.680952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" event={"ID":"baba2139-d918-4f6f-9eab-3d14183d9659","Type":"ContainerStarted","Data":"73d006ed6490f0403a26e05168d3ce273436246d0caabc372d9f8db3d5eb6a75"} Jul 30 10:58:20.681554 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:20.681267 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" Jul 30 10:58:20.683033 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:20.683000 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" event={"ID":"2d7944ed-a849-430c-b6eb-bb8bd57754c0","Type":"ContainerStarted","Data":"55c7d81f148f4177f62354740fca7946c65c99444e462c66c8d555a47c8470be"} Jul 30 10:58:20.683285 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:20.683267 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" Jul 30 10:58:20.697944 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:20.697890 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" podStartSLOduration=1.637848043 podStartE2EDuration="3.697872367s" totalImagesPullingTime="2.060024324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:58:20.695748013 +0000 UTC m=+608.668609049" watchObservedRunningTime="2026-07-30 10:58:20.697872367 +0000 UTC m=+608.670733380" Jul 30 10:58:20.712963 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:20.712904 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" podStartSLOduration=1.6026145729999999 podStartE2EDuration="3.712885553s" totalImagesPullingTime="2.11027098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:58:20.710331103 +0000 UTC m=+608.683192161" watchObservedRunningTime="2026-07-30 10:58:20.712885553 +0000 UTC m=+608.685746588" Jul 30 10:58:25.689356 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:25.689327 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-669686cd4-rhjj8" Jul 30 10:58:31.691234 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:31.691199 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-w2xrv" Jul 30 10:58:31.691692 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:58:31.691473 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-h8q2d" Jul 30 10:59:00.096344 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.096309 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc"] Jul 30 10:59:00.102758 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.102722 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.106556 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.106526 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 30 10:59:00.106904 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.106742 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f52tt\"" Jul 30 10:59:00.106904 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.106754 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 30 10:59:00.106904 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.106752 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 30 10:59:00.115610 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.115581 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc"] Jul 30 10:59:00.158216 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.158185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmpbx\" (UniqueName: \"kubernetes.io/projected/87b51936-8941-41c4-8356-051041617c5a-kube-api-access-vmpbx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.158419 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.158224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.158419 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.158297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b51936-8941-41c4-8356-051041617c5a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.158419 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.158335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.158419 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.158363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.158419 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.158395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.259927 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.259895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.259927 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.259930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.260260 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.259961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.260260 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.260037 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmpbx\" (UniqueName: \"kubernetes.io/projected/87b51936-8941-41c4-8356-051041617c5a-kube-api-access-vmpbx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.260260 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.260071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.260260 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.260135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b51936-8941-41c4-8356-051041617c5a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.260503 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.260427 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.260503 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.260491 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.260601 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.260525 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.262280 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.262261 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87b51936-8941-41c4-8356-051041617c5a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.262537 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.262519 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87b51936-8941-41c4-8356-051041617c5a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.268858 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.268833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmpbx\" (UniqueName: \"kubernetes.io/projected/87b51936-8941-41c4-8356-051041617c5a-kube-api-access-vmpbx\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pdqjc\" (UID: \"87b51936-8941-41c4-8356-051041617c5a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.418955 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.418857 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:00.575363 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.575334 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc"] Jul 30 10:59:00.575702 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:59:00.575681 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b51936_8941_41c4_8356_051041617c5a.slice/crio-d908b6e8c66f80082cc4c5707f01bf6a004842c4ce858843668bfee0f4add032 WatchSource:0}: Error finding container d908b6e8c66f80082cc4c5707f01bf6a004842c4ce858843668bfee0f4add032: Status 404 returned error can't find the container with id d908b6e8c66f80082cc4c5707f01bf6a004842c4ce858843668bfee0f4add032 Jul 30 10:59:00.577059 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.577041 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 10:59:00.916540 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:00.916445 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" event={"ID":"87b51936-8941-41c4-8356-051041617c5a","Type":"ContainerStarted","Data":"d908b6e8c66f80082cc4c5707f01bf6a004842c4ce858843668bfee0f4add032"} Jul 30 10:59:01.923423 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:01.923386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" event={"ID":"87b51936-8941-41c4-8356-051041617c5a","Type":"ContainerStarted","Data":"440572595eacd5bb1496a1f51c71dec703ae5720472af7b86c835692caed62f7"} Jul 30 10:59:01.948036 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:01.947920 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" podStartSLOduration=0.828597897 podStartE2EDuration="1.947899037s" totalImagesPullingTime="1.11930114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:59:01.944279152 +0000 UTC m=+649.917140187" watchObservedRunningTime="2026-07-30 10:59:01.947899037 +0000 UTC m=+649.920760074" Jul 30 10:59:10.419200 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:10.419149 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:10.419622 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:10.419238 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:10.432556 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:10.432534 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:10.987674 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:10.987518 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pdqjc" Jul 30 10:59:37.793421 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.793384 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk"] Jul 30 10:59:37.798480 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.798457 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.800673 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.800651 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 30 10:59:37.805580 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.805555 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk"] Jul 30 10:59:37.809593 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.809563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.809723 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.809632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.809723 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.809661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.809723 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.809700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506eba34-1d5d-4afd-8447-d1d15354a095-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.809897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.809768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nhst\" (UniqueName: \"kubernetes.io/projected/506eba34-1d5d-4afd-8447-d1d15354a095-kube-api-access-7nhst\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.809897 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.809844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.910819 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.910784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.910999 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.910851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.910999 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.910886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.910999 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.910905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.910999 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.910927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506eba34-1d5d-4afd-8447-d1d15354a095-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.910999 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.910948 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nhst\" (UniqueName: \"kubernetes.io/projected/506eba34-1d5d-4afd-8447-d1d15354a095-kube-api-access-7nhst\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.911340 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.911308 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.911340 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.911323 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.911499 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.911386 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.913194 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.913175 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/506eba34-1d5d-4afd-8447-d1d15354a095-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.913486 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.913465 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/506eba34-1d5d-4afd-8447-d1d15354a095-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:37.918904 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:37.918874 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nhst\" (UniqueName: \"kubernetes.io/projected/506eba34-1d5d-4afd-8447-d1d15354a095-kube-api-access-7nhst\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk\" (UID: \"506eba34-1d5d-4afd-8447-d1d15354a095\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:38.111202 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:38.111073 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:38.252919 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:38.252892 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk"] Jul 30 10:59:38.254188 ip-10-0-138-131 kubenswrapper[2575]: W0730 10:59:38.254151 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506eba34_1d5d_4afd_8447_d1d15354a095.slice/crio-d5e1d036f0fcf1e7f93ef3eda7ad4793a7b3a0fe3ef0a687043f0c58a3bbc686 WatchSource:0}: Error finding container d5e1d036f0fcf1e7f93ef3eda7ad4793a7b3a0fe3ef0a687043f0c58a3bbc686: Status 404 returned error can't find the container with id d5e1d036f0fcf1e7f93ef3eda7ad4793a7b3a0fe3ef0a687043f0c58a3bbc686 Jul 30 10:59:39.138054 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:39.138019 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" event={"ID":"506eba34-1d5d-4afd-8447-d1d15354a095","Type":"ContainerStarted","Data":"d5e1d036f0fcf1e7f93ef3eda7ad4793a7b3a0fe3ef0a687043f0c58a3bbc686"} Jul 30 10:59:41.151445 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:41.151409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" event={"ID":"506eba34-1d5d-4afd-8447-d1d15354a095","Type":"ContainerStarted","Data":"c0de0e6d79175a8c9d08f5931c6096ea14c3486a7f4e6f0002a25a181622cbd6"} Jul 30 10:59:41.171707 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:41.171654 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" podStartSLOduration=2.342678004 podStartE2EDuration="4.171634933s" totalImagesPullingTime="1.828956929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 10:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 10:59:41.170028087 +0000 UTC m=+689.142889151" watchObservedRunningTime="2026-07-30 10:59:41.171634933 +0000 UTC m=+689.144495968" Jul 30 10:59:48.112304 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:48.112261 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:48.112881 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:48.112343 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:48.125081 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:48.125055 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 10:59:48.209103 ip-10-0-138-131 kubenswrapper[2575]: I0730 10:59:48.209066 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk" Jul 30 11:00:00.142420 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:00.142384 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2"] Jul 30 11:00:00.146331 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:00.146313 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2" Jul 30 11:00:00.148649 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:00.148630 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:00:00.152786 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:00.152759 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2"] Jul 30 11:00:00.158128 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:00.158108 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2" Jul 30 11:00:00.305477 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:00.305444 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2"] Jul 30 11:00:00.307661 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:00:00.307631 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e30908c_70a8_4a62_82b5_d7571ea4bc70.slice/crio-e3f94b49e7d55959112dd2aebee90fbe8829a38ba0af2ab9796d221fbe797bea WatchSource:0}: Error finding container e3f94b49e7d55959112dd2aebee90fbe8829a38ba0af2ab9796d221fbe797bea: Status 404 returned error can't find the container with id e3f94b49e7d55959112dd2aebee90fbe8829a38ba0af2ab9796d221fbe797bea Jul 30 11:00:01.275290 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:01.275254 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2" event={"ID":"7e30908c-70a8-4a62-82b5-d7571ea4bc70","Type":"ContainerStarted","Data":"e3f94b49e7d55959112dd2aebee90fbe8829a38ba0af2ab9796d221fbe797bea"} Jul 30 11:00:03.289963 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:03.289930 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2" event={"ID":"7e30908c-70a8-4a62-82b5-d7571ea4bc70","Type":"ContainerStarted","Data":"eee7ae94fd3b588fee82b8cfa072f9dbb4992a29743c038c6b166458f1511b48"} Jul 30 11:00:03.306783 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:03.306674 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2" podStartSLOduration=0.826199206 podStartE2EDuration="3.306658574s" totalImagesPullingTime="2.480459368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:00:03.305449406 +0000 UTC m=+711.278310469" watchObservedRunningTime="2026-07-30 11:00:03.306658574 +0000 UTC m=+711.279519695" Jul 30 11:00:04.297736 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:04.297704 2575 generic.go:356] "Generic (PLEG): container finished" podID="7e30908c-70a8-4a62-82b5-d7571ea4bc70" containerID="eee7ae94fd3b588fee82b8cfa072f9dbb4992a29743c038c6b166458f1511b48" exitCode=0 Jul 30 11:00:04.298163 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:04.297802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2" event={"ID":"7e30908c-70a8-4a62-82b5-d7571ea4bc70","Type":"ContainerDied","Data":"eee7ae94fd3b588fee82b8cfa072f9dbb4992a29743c038c6b166458f1511b48"} Jul 30 11:00:05.447972 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:05.447945 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2" Jul 30 11:00:06.310784 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:06.310756 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2" Jul 30 11:00:06.310986 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:06.310791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2" event={"ID":"7e30908c-70a8-4a62-82b5-d7571ea4bc70","Type":"ContainerDied","Data":"e3f94b49e7d55959112dd2aebee90fbe8829a38ba0af2ab9796d221fbe797bea"} Jul 30 11:00:06.310986 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:00:06.310842 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3f94b49e7d55959112dd2aebee90fbe8829a38ba0af2ab9796d221fbe797bea" Jul 30 11:01:54.644991 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:54.644949 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-2fkmh"] Jul 30 11:01:54.645515 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:54.645195 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" podUID="f734ed02-8f10-4332-8d79-f77521c55ccc" containerName="manager" containerID="cri-o://bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd" gracePeriod=30 Jul 30 11:01:54.908748 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:54.908674 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 11:01:54.952890 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:54.952850 2575 generic.go:356] "Generic (PLEG): container finished" podID="f734ed02-8f10-4332-8d79-f77521c55ccc" containerID="bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd" exitCode=0 Jul 30 11:01:54.953065 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:54.952900 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" Jul 30 11:01:54.953065 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:54.952932 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" event={"ID":"f734ed02-8f10-4332-8d79-f77521c55ccc","Type":"ContainerDied","Data":"bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd"} Jul 30 11:01:54.953065 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:54.952989 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d558b5f66-2fkmh" event={"ID":"f734ed02-8f10-4332-8d79-f77521c55ccc","Type":"ContainerDied","Data":"7f04b57cbdc6199bcc9ff4cee97e893b170b375d206fae0e276b76240e63de1e"} Jul 30 11:01:54.953065 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:54.953020 2575 scope.go:117] "RemoveContainer" containerID="bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd" Jul 30 11:01:54.962581 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:54.962560 2575 scope.go:117] "RemoveContainer" containerID="bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd" Jul 30 11:01:54.962903 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:01:54.962877 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd\": container with ID starting with bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd not found: ID does not exist" containerID="bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd" Jul 30 11:01:54.963005 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:54.962914 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd"} err="failed to get container status \"bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd\": rpc error: code = NotFound desc = could not find container \"bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd\": container with ID starting with bea109c64c336365d609d0395eab418748df8af44b6cd5e59f8028495bb730dd not found: ID does not exist" Jul 30 11:01:55.030435 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:55.030400 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4479c\" (UniqueName: \"kubernetes.io/projected/f734ed02-8f10-4332-8d79-f77521c55ccc-kube-api-access-4479c\") pod \"f734ed02-8f10-4332-8d79-f77521c55ccc\" (UID: \"f734ed02-8f10-4332-8d79-f77521c55ccc\") " Jul 30 11:01:55.030642 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:55.030450 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f734ed02-8f10-4332-8d79-f77521c55ccc-webhook-cert\") pod \"f734ed02-8f10-4332-8d79-f77521c55ccc\" (UID: \"f734ed02-8f10-4332-8d79-f77521c55ccc\") " Jul 30 11:01:55.032586 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:55.032552 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f734ed02-8f10-4332-8d79-f77521c55ccc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f734ed02-8f10-4332-8d79-f77521c55ccc" (UID: "f734ed02-8f10-4332-8d79-f77521c55ccc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 11:01:55.032712 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:55.032606 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f734ed02-8f10-4332-8d79-f77521c55ccc-kube-api-access-4479c" (OuterVolumeSpecName: "kube-api-access-4479c") pod "f734ed02-8f10-4332-8d79-f77521c55ccc" (UID: "f734ed02-8f10-4332-8d79-f77521c55ccc"). InnerVolumeSpecName "kube-api-access-4479c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:01:55.131961 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:55.131924 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4479c\" (UniqueName: \"kubernetes.io/projected/f734ed02-8f10-4332-8d79-f77521c55ccc-kube-api-access-4479c\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:01:55.131961 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:55.131954 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f734ed02-8f10-4332-8d79-f77521c55ccc-webhook-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:01:55.275737 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:55.275704 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-2fkmh"] Jul 30 11:01:55.280185 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:55.280155 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-2fkmh"] Jul 30 11:01:56.527450 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:01:56.527417 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f734ed02-8f10-4332-8d79-f77521c55ccc" path="/var/lib/kubelet/pods/f734ed02-8f10-4332-8d79-f77521c55ccc/volumes" Jul 30 11:02:56.074573 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.074538 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d558b5f66-nv8mw"] Jul 30 11:02:56.074996 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.074969 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e30908c-70a8-4a62-82b5-d7571ea4bc70" containerName="cleanup" Jul 30 11:02:56.074996 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.074980 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e30908c-70a8-4a62-82b5-d7571ea4bc70" containerName="cleanup" Jul 30 11:02:56.074996 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.074994 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f734ed02-8f10-4332-8d79-f77521c55ccc" containerName="manager" Jul 30 11:02:56.075117 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.075000 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f734ed02-8f10-4332-8d79-f77521c55ccc" containerName="manager" Jul 30 11:02:56.075117 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.075066 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e30908c-70a8-4a62-82b5-d7571ea4bc70" containerName="cleanup" Jul 30 11:02:56.075117 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.075076 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f734ed02-8f10-4332-8d79-f77521c55ccc" containerName="manager" Jul 30 11:02:56.078245 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.078229 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:02:56.080169 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.080143 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 11:02:56.080664 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.080645 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ghk2x\"" Jul 30 11:02:56.080732 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.080659 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 11:02:56.088041 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.088015 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-nv8mw"] Jul 30 11:02:56.119831 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.119789 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsf6n\" (UniqueName: \"kubernetes.io/projected/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-kube-api-access-wsf6n\") pod \"maas-controller-6d558b5f66-nv8mw\" (UID: \"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8\") " pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:02:56.120022 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.119923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-webhook-cert\") pod \"maas-controller-6d558b5f66-nv8mw\" (UID: \"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8\") " pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:02:56.221329 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.221288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-webhook-cert\") pod \"maas-controller-6d558b5f66-nv8mw\" (UID: \"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8\") " pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:02:56.221499 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.221389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsf6n\" (UniqueName: \"kubernetes.io/projected/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-kube-api-access-wsf6n\") pod \"maas-controller-6d558b5f66-nv8mw\" (UID: \"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8\") " pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:02:56.223798 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.223766 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-webhook-cert\") pod \"maas-controller-6d558b5f66-nv8mw\" (UID: \"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8\") " pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:02:56.229598 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.229570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsf6n\" (UniqueName: \"kubernetes.io/projected/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-kube-api-access-wsf6n\") pod \"maas-controller-6d558b5f66-nv8mw\" (UID: \"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8\") " pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:02:56.389248 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.389154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:02:56.530783 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:56.530747 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-nv8mw"] Jul 30 11:02:56.532011 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:02:56.531987 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7fb473_3e5b_4b2a_ae01_b0d20432b4e8.slice/crio-5d25526cd99c47375dce08ab7765b6fb6db0346bbec6cd062023cff319aa2792 WatchSource:0}: Error finding container 5d25526cd99c47375dce08ab7765b6fb6db0346bbec6cd062023cff319aa2792: Status 404 returned error can't find the container with id 5d25526cd99c47375dce08ab7765b6fb6db0346bbec6cd062023cff319aa2792 Jul 30 11:02:57.328129 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:57.328075 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" event={"ID":"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8","Type":"ContainerStarted","Data":"4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f"} Jul 30 11:02:57.328602 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:57.328299 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:02:57.328602 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:57.328368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" event={"ID":"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8","Type":"ContainerStarted","Data":"5d25526cd99c47375dce08ab7765b6fb6db0346bbec6cd062023cff319aa2792"} Jul 30 11:02:57.347718 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:02:57.347661 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" podStartSLOduration=0.869113903 podStartE2EDuration="1.347644203s" totalImagesPullingTime="478.5303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:02:57.345189245 +0000 UTC m=+885.318050293" watchObservedRunningTime="2026-07-30 11:02:57.347644203 +0000 UTC m=+885.320505238" Jul 30 11:03:08.335624 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:08.335548 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:03:12.494009 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:12.493978 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:03:12.494553 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:12.494314 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:03:12.981839 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:12.981799 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-nv8mw"] Jul 30 11:03:12.982706 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:12.982668 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" podUID="6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8" containerName="manager" containerID="cri-o://4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f" gracePeriod=30 Jul 30 11:03:13.243586 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.243517 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:03:13.387078 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.387044 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsf6n\" (UniqueName: \"kubernetes.io/projected/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-kube-api-access-wsf6n\") pod \"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8\" (UID: \"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8\") " Jul 30 11:03:13.387305 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.387190 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-webhook-cert\") pod \"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8\" (UID: \"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8\") " Jul 30 11:03:13.389250 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.389217 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8" (UID: "6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 11:03:13.389367 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.389263 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-kube-api-access-wsf6n" (OuterVolumeSpecName: "kube-api-access-wsf6n") pod "6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8" (UID: "6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8"). InnerVolumeSpecName "kube-api-access-wsf6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:03:13.422779 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.422740 2575 generic.go:356] "Generic (PLEG): container finished" podID="6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8" containerID="4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f" exitCode=0 Jul 30 11:03:13.422969 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.422783 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" Jul 30 11:03:13.422969 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.422820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" event={"ID":"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8","Type":"ContainerDied","Data":"4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f"} Jul 30 11:03:13.422969 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.422870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d558b5f66-nv8mw" event={"ID":"6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8","Type":"ContainerDied","Data":"5d25526cd99c47375dce08ab7765b6fb6db0346bbec6cd062023cff319aa2792"} Jul 30 11:03:13.422969 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.422893 2575 scope.go:117] "RemoveContainer" containerID="4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f" Jul 30 11:03:13.432837 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.432821 2575 scope.go:117] "RemoveContainer" containerID="4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f" Jul 30 11:03:13.433150 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:03:13.433132 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f\": container with ID starting with 4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f not found: ID does not exist" containerID="4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f" Jul 30 11:03:13.433198 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.433159 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f"} err="failed to get container status \"4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f\": rpc error: code = NotFound desc = could not find container \"4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f\": container with ID starting with 4d6e097d44b36b3972139e3806ccc0633bcd6abf2d920c0d06e990bc60765f7f not found: ID does not exist" Jul 30 11:03:13.445128 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.445103 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-nv8mw"] Jul 30 11:03:13.448699 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.448674 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-nv8mw"] Jul 30 11:03:13.488032 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.488009 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-webhook-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:03:13.488032 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:13.488030 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsf6n\" (UniqueName: \"kubernetes.io/projected/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8-kube-api-access-wsf6n\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:03:14.242361 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.242317 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d558b5f66-q6g9n"] Jul 30 11:03:14.242772 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.242759 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8" containerName="manager" Jul 30 11:03:14.242772 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.242771 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8" containerName="manager" Jul 30 11:03:14.242863 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.242857 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8" containerName="manager" Jul 30 11:03:14.247129 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.247111 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d558b5f66-q6g9n" Jul 30 11:03:14.249882 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.249865 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 30 11:03:14.250009 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.249883 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 30 11:03:14.250009 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.249880 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ghk2x\"" Jul 30 11:03:14.257620 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.257591 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-q6g9n"] Jul 30 11:03:14.396069 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.396028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3ec4cdd-0fa1-4009-914d-5f4b5e119024-webhook-cert\") pod \"maas-controller-6d558b5f66-q6g9n\" (UID: \"b3ec4cdd-0fa1-4009-914d-5f4b5e119024\") " pod="opendatahub/maas-controller-6d558b5f66-q6g9n" Jul 30 11:03:14.396289 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.396189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvj74\" (UniqueName: \"kubernetes.io/projected/b3ec4cdd-0fa1-4009-914d-5f4b5e119024-kube-api-access-rvj74\") pod \"maas-controller-6d558b5f66-q6g9n\" (UID: \"b3ec4cdd-0fa1-4009-914d-5f4b5e119024\") " pod="opendatahub/maas-controller-6d558b5f66-q6g9n" Jul 30 11:03:14.496825 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.496728 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvj74\" (UniqueName: \"kubernetes.io/projected/b3ec4cdd-0fa1-4009-914d-5f4b5e119024-kube-api-access-rvj74\") pod \"maas-controller-6d558b5f66-q6g9n\" (UID: \"b3ec4cdd-0fa1-4009-914d-5f4b5e119024\") " pod="opendatahub/maas-controller-6d558b5f66-q6g9n" Jul 30 11:03:14.496825 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.496793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3ec4cdd-0fa1-4009-914d-5f4b5e119024-webhook-cert\") pod \"maas-controller-6d558b5f66-q6g9n\" (UID: \"b3ec4cdd-0fa1-4009-914d-5f4b5e119024\") " pod="opendatahub/maas-controller-6d558b5f66-q6g9n" Jul 30 11:03:14.499207 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.499174 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3ec4cdd-0fa1-4009-914d-5f4b5e119024-webhook-cert\") pod \"maas-controller-6d558b5f66-q6g9n\" (UID: \"b3ec4cdd-0fa1-4009-914d-5f4b5e119024\") " pod="opendatahub/maas-controller-6d558b5f66-q6g9n" Jul 30 11:03:14.504327 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.504295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvj74\" (UniqueName: \"kubernetes.io/projected/b3ec4cdd-0fa1-4009-914d-5f4b5e119024-kube-api-access-rvj74\") pod \"maas-controller-6d558b5f66-q6g9n\" (UID: \"b3ec4cdd-0fa1-4009-914d-5f4b5e119024\") " pod="opendatahub/maas-controller-6d558b5f66-q6g9n" Jul 30 11:03:14.528187 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.528154 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8" path="/var/lib/kubelet/pods/6e7fb473-3e5b-4b2a-ae01-b0d20432b4e8/volumes" Jul 30 11:03:14.559318 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.559279 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d558b5f66-q6g9n" Jul 30 11:03:14.709635 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:14.709516 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d558b5f66-q6g9n"] Jul 30 11:03:14.712014 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:03:14.711980 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ec4cdd_0fa1_4009_914d_5f4b5e119024.slice/crio-59aa3816a85eb1a4d896901cae8415f0c8b1b58df46a7254405d28c422c7d440 WatchSource:0}: Error finding container 59aa3816a85eb1a4d896901cae8415f0c8b1b58df46a7254405d28c422c7d440: Status 404 returned error can't find the container with id 59aa3816a85eb1a4d896901cae8415f0c8b1b58df46a7254405d28c422c7d440 Jul 30 11:03:15.438255 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:15.438165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d558b5f66-q6g9n" event={"ID":"b3ec4cdd-0fa1-4009-914d-5f4b5e119024","Type":"ContainerStarted","Data":"9bb3b1d70ab4e780df474582a46ad500ebea9a9fd700beda330f68b9e6cd0c43"} Jul 30 11:03:15.438255 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:15.438214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d558b5f66-q6g9n" event={"ID":"b3ec4cdd-0fa1-4009-914d-5f4b5e119024","Type":"ContainerStarted","Data":"59aa3816a85eb1a4d896901cae8415f0c8b1b58df46a7254405d28c422c7d440"} Jul 30 11:03:15.438645 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:15.438269 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d558b5f66-q6g9n" Jul 30 11:03:15.455821 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:15.455776 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d558b5f66-q6g9n" podStartSLOduration=1.010386439 podStartE2EDuration="1.455757938s" totalImagesPullingTime="445.371499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:03:15.454360712 +0000 UTC m=+903.427221768" watchObservedRunningTime="2026-07-30 11:03:15.455757938 +0000 UTC m=+903.428619026" Jul 30 11:03:26.446247 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:03:26.446214 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d558b5f66-q6g9n" Jul 30 11:08:12.534574 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:08:12.534544 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:08:12.536650 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:08:12.536625 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:09:43.970280 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:43.970245 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw"] Jul 30 11:09:43.970815 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:43.970461 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" podUID="3057b1fb-9c43-4ef9-90e5-e82086ebd8c0" containerName="manager" containerID="cri-o://0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b" gracePeriod=10 Jul 30 11:09:44.233931 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.233867 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 11:09:44.305874 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.305839 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-extensions-socket-volume\") pod \"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0\" (UID: \"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0\") " Jul 30 11:09:44.306040 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.305976 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dq2n\" (UniqueName: \"kubernetes.io/projected/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-kube-api-access-4dq2n\") pod \"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0\" (UID: \"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0\") " Jul 30 11:09:44.306307 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.306285 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3057b1fb-9c43-4ef9-90e5-e82086ebd8c0" (UID: "3057b1fb-9c43-4ef9-90e5-e82086ebd8c0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:09:44.308136 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.308114 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-kube-api-access-4dq2n" (OuterVolumeSpecName: "kube-api-access-4dq2n") pod "3057b1fb-9c43-4ef9-90e5-e82086ebd8c0" (UID: "3057b1fb-9c43-4ef9-90e5-e82086ebd8c0"). InnerVolumeSpecName "kube-api-access-4dq2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:09:44.407271 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.407236 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-extensions-socket-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:09:44.407271 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.407267 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dq2n\" (UniqueName: \"kubernetes.io/projected/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0-kube-api-access-4dq2n\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:09:44.782974 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.782942 2575 generic.go:356] "Generic (PLEG): container finished" podID="3057b1fb-9c43-4ef9-90e5-e82086ebd8c0" containerID="0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b" exitCode=0 Jul 30 11:09:44.783191 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.782987 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" Jul 30 11:09:44.783191 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.783028 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" event={"ID":"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0","Type":"ContainerDied","Data":"0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b"} Jul 30 11:09:44.783191 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.783100 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw" event={"ID":"3057b1fb-9c43-4ef9-90e5-e82086ebd8c0","Type":"ContainerDied","Data":"a6e25172fc0e8eaa5e8ab90663dda904a77f33fb85e04e6125e048bd3d29c828"} Jul 30 11:09:44.783191 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.783112 2575 scope.go:117] "RemoveContainer" containerID="0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b" Jul 30 11:09:44.794313 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.794292 2575 scope.go:117] "RemoveContainer" containerID="0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b" Jul 30 11:09:44.794556 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:09:44.794533 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b\": container with ID starting with 0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b not found: ID does not exist" containerID="0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b" Jul 30 11:09:44.794625 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.794569 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b"} err="failed to get container status \"0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b\": rpc error: code = NotFound desc = could not find container \"0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b\": container with ID starting with 0acb2c2f99aace0bfe5f2abce1c84f5d8bda41ac4a9b49ba9c0a7e26fe7dc51b not found: ID does not exist" Jul 30 11:09:44.800330 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.800305 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw"] Jul 30 11:09:44.803405 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:44.803386 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-v6czw"] Jul 30 11:09:46.528060 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:09:46.528026 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3057b1fb-9c43-4ef9-90e5-e82086ebd8c0" path="/var/lib/kubelet/pods/3057b1fb-9c43-4ef9-90e5-e82086ebd8c0/volumes" Jul 30 11:10:44.686431 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.686345 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt"] Jul 30 11:10:44.686908 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.686783 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3057b1fb-9c43-4ef9-90e5-e82086ebd8c0" containerName="manager" Jul 30 11:10:44.686908 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.686794 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3057b1fb-9c43-4ef9-90e5-e82086ebd8c0" containerName="manager" Jul 30 11:10:44.686908 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.686859 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3057b1fb-9c43-4ef9-90e5-e82086ebd8c0" containerName="manager" Jul 30 11:10:44.690016 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.689998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:10:44.692262 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.692240 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fqxbm\"" Jul 30 11:10:44.700703 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.700677 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt"] Jul 30 11:10:44.782212 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.782161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/272bf8cb-ec29-459e-92f3-e2af3f34415c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-kxfvt\" (UID: \"272bf8cb-ec29-459e-92f3-e2af3f34415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:10:44.782395 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.782269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggmbs\" (UniqueName: \"kubernetes.io/projected/272bf8cb-ec29-459e-92f3-e2af3f34415c-kube-api-access-ggmbs\") pod \"kuadrant-operator-controller-manager-68cf589589-kxfvt\" (UID: \"272bf8cb-ec29-459e-92f3-e2af3f34415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:10:44.883216 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.883169 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/272bf8cb-ec29-459e-92f3-e2af3f34415c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-kxfvt\" (UID: \"272bf8cb-ec29-459e-92f3-e2af3f34415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:10:44.883393 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.883273 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmbs\" (UniqueName: \"kubernetes.io/projected/272bf8cb-ec29-459e-92f3-e2af3f34415c-kube-api-access-ggmbs\") pod \"kuadrant-operator-controller-manager-68cf589589-kxfvt\" (UID: \"272bf8cb-ec29-459e-92f3-e2af3f34415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:10:44.883593 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.883574 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/272bf8cb-ec29-459e-92f3-e2af3f34415c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-kxfvt\" (UID: \"272bf8cb-ec29-459e-92f3-e2af3f34415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:10:44.890972 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:44.890952 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggmbs\" (UniqueName: \"kubernetes.io/projected/272bf8cb-ec29-459e-92f3-e2af3f34415c-kube-api-access-ggmbs\") pod \"kuadrant-operator-controller-manager-68cf589589-kxfvt\" (UID: \"272bf8cb-ec29-459e-92f3-e2af3f34415c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:10:45.001038 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:45.000935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:10:45.155394 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:45.155369 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt"] Jul 30 11:10:45.157274 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:10:45.157241 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272bf8cb_ec29_459e_92f3_e2af3f34415c.slice/crio-7583f7bd5963cba23e2e9735090f29a8cb9618f79c81caee6ddf75357eb9bd77 WatchSource:0}: Error finding container 7583f7bd5963cba23e2e9735090f29a8cb9618f79c81caee6ddf75357eb9bd77: Status 404 returned error can't find the container with id 7583f7bd5963cba23e2e9735090f29a8cb9618f79c81caee6ddf75357eb9bd77 Jul 30 11:10:46.145683 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:46.145649 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" event={"ID":"272bf8cb-ec29-459e-92f3-e2af3f34415c","Type":"ContainerStarted","Data":"1a7030596ce875d4fefc98bb049cc2a1e957a24477aecb3e1e3230f7db476184"} Jul 30 11:10:46.146113 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:46.145708 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" event={"ID":"272bf8cb-ec29-459e-92f3-e2af3f34415c","Type":"ContainerStarted","Data":"7583f7bd5963cba23e2e9735090f29a8cb9618f79c81caee6ddf75357eb9bd77"} Jul 30 11:10:46.146113 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:46.145733 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:10:46.175380 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:46.175326 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" podStartSLOduration=2.175307546 podStartE2EDuration="2.175307546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:10:46.171939962 +0000 UTC m=+1354.144800997" watchObservedRunningTime="2026-07-30 11:10:46.175307546 +0000 UTC m=+1354.148168578" Jul 30 11:10:57.153705 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:10:57.153668 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:13:12.573462 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:13:12.573436 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:13:12.576367 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:13:12.576347 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:15:00.183928 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:00.183894 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b"] Jul 30 11:15:00.187952 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:00.187934 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b" Jul 30 11:15:00.195234 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:00.195212 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:15:00.197746 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:00.197731 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b" Jul 30 11:15:00.208986 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:00.208963 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b"] Jul 30 11:15:00.347801 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:00.347774 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b"] Jul 30 11:15:00.349394 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:15:00.349369 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04665bef_eab4_49d5_b506_0298470c6382.slice/crio-d65ff82e3b8959b8eb162af81dec4080ae46a05a02ba923beb7c13bb78428b9e WatchSource:0}: Error finding container d65ff82e3b8959b8eb162af81dec4080ae46a05a02ba923beb7c13bb78428b9e: Status 404 returned error can't find the container with id d65ff82e3b8959b8eb162af81dec4080ae46a05a02ba923beb7c13bb78428b9e Jul 30 11:15:00.635597 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:00.635561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b" event={"ID":"04665bef-eab4-49d5-b506-0298470c6382","Type":"ContainerStarted","Data":"6bf025c11229357656c854f561d5f97e701e68175c9fa321759f291ac139edc7"} Jul 30 11:15:00.635778 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:00.635617 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b" event={"ID":"04665bef-eab4-49d5-b506-0298470c6382","Type":"ContainerStarted","Data":"d65ff82e3b8959b8eb162af81dec4080ae46a05a02ba923beb7c13bb78428b9e"} Jul 30 11:15:00.654653 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:00.654607 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b" podStartSLOduration=0.654593413 podStartE2EDuration="654.593413ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:15:00.653467862 +0000 UTC m=+1608.626328907" watchObservedRunningTime="2026-07-30 11:15:00.654593413 +0000 UTC m=+1608.627454447" Jul 30 11:15:01.642968 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:01.642932 2575 generic.go:356] "Generic (PLEG): container finished" podID="04665bef-eab4-49d5-b506-0298470c6382" containerID="6bf025c11229357656c854f561d5f97e701e68175c9fa321759f291ac139edc7" exitCode=0 Jul 30 11:15:01.643368 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:01.642998 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b" event={"ID":"04665bef-eab4-49d5-b506-0298470c6382","Type":"ContainerDied","Data":"6bf025c11229357656c854f561d5f97e701e68175c9fa321759f291ac139edc7"} Jul 30 11:15:02.800146 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:02.800120 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b" Jul 30 11:15:03.655720 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:03.655634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b" event={"ID":"04665bef-eab4-49d5-b506-0298470c6382","Type":"ContainerDied","Data":"d65ff82e3b8959b8eb162af81dec4080ae46a05a02ba923beb7c13bb78428b9e"} Jul 30 11:15:03.655720 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:03.655670 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756835-wl69b" Jul 30 11:15:03.655720 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:15:03.655690 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d65ff82e3b8959b8eb162af81dec4080ae46a05a02ba923beb7c13bb78428b9e" Jul 30 11:18:12.611741 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:18:12.611712 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:18:12.616424 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:18:12.616398 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:19:09.016912 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.016881 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw"] Jul 30 11:19:09.017392 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.017309 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04665bef-eab4-49d5-b506-0298470c6382" containerName="cleanup" Jul 30 11:19:09.017392 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.017320 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="04665bef-eab4-49d5-b506-0298470c6382" containerName="cleanup" Jul 30 11:19:09.017466 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.017406 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="04665bef-eab4-49d5-b506-0298470c6382" containerName="cleanup" Jul 30 11:19:09.020602 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.020581 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.022444 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.022423 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2ca37a8f-serving-cert\"" Jul 30 11:19:09.030631 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.030610 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw"] Jul 30 11:19:09.213882 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.213845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twcjn\" (UniqueName: \"kubernetes.io/projected/3617fe7f-0f1c-494b-8edb-9648ab127547-kube-api-access-twcjn\") pod \"maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.213882 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.213887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3617fe7f-0f1c-494b-8edb-9648ab127547-maas-api-tls\") pod \"maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.214151 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.213935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3617fe7f-0f1c-494b-8edb-9648ab127547-openshift-service-ca\") pod \"maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.314652 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.314621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3617fe7f-0f1c-494b-8edb-9648ab127547-openshift-service-ca\") pod \"maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.314792 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.314713 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twcjn\" (UniqueName: \"kubernetes.io/projected/3617fe7f-0f1c-494b-8edb-9648ab127547-kube-api-access-twcjn\") pod \"maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.314792 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.314746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3617fe7f-0f1c-494b-8edb-9648ab127547-maas-api-tls\") pod \"maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.315273 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.315248 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3617fe7f-0f1c-494b-8edb-9648ab127547-openshift-service-ca\") pod \"maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.317179 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.317162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3617fe7f-0f1c-494b-8edb-9648ab127547-maas-api-tls\") pod \"maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.321805 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.321787 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twcjn\" (UniqueName: \"kubernetes.io/projected/3617fe7f-0f1c-494b-8edb-9648ab127547-kube-api-access-twcjn\") pod \"maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.333416 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.333395 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:09.476468 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.476445 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw"] Jul 30 11:19:09.479120 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:19:09.479075 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3617fe7f_0f1c_494b_8edb_9648ab127547.slice/crio-7bb53cba9f8a96703a7ba4000b8acf7f9f4f42465e7b9c4753efde3277dcf4b4 WatchSource:0}: Error finding container 7bb53cba9f8a96703a7ba4000b8acf7f9f4f42465e7b9c4753efde3277dcf4b4: Status 404 returned error can't find the container with id 7bb53cba9f8a96703a7ba4000b8acf7f9f4f42465e7b9c4753efde3277dcf4b4 Jul 30 11:19:09.480460 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.480440 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 11:19:09.642436 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.642361 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k"] Jul 30 11:19:09.647512 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.647492 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:09.649397 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.649375 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2ca37a8f\"" Jul 30 11:19:09.655754 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.655727 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k"] Jul 30 11:19:09.686491 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.686467 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m"] Jul 30 11:19:09.690016 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.690001 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:09.691790 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.691773 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2ca37a8f-dockercfg-bpfz2\"" Jul 30 11:19:09.700603 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.700576 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m"] Jul 30 11:19:09.717646 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.717616 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfck5\" (UniqueName: \"kubernetes.io/projected/51fcbefe-e2d7-4127-a4f4-124dbb68c405-kube-api-access-rfck5\") pod \"payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k\" (UID: \"51fcbefe-e2d7-4127-a4f4-124dbb68c405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:09.717925 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.717906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fcbefe-e2d7-4127-a4f4-124dbb68c405-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k\" (UID: \"51fcbefe-e2d7-4127-a4f4-124dbb68c405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:09.818643 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.818602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fcbefe-e2d7-4127-a4f4-124dbb68c405-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k\" (UID: \"51fcbefe-e2d7-4127-a4f4-124dbb68c405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:09.818849 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.818677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnp4l\" (UniqueName: \"kubernetes.io/projected/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-kube-api-access-gnp4l\") pod \"payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m\" (UID: \"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21\") " pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:09.818849 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.818715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfck5\" (UniqueName: \"kubernetes.io/projected/51fcbefe-e2d7-4127-a4f4-124dbb68c405-kube-api-access-rfck5\") pod \"payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k\" (UID: \"51fcbefe-e2d7-4127-a4f4-124dbb68c405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:09.818849 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.818747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m\" (UID: \"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21\") " pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:09.819456 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.819434 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fcbefe-e2d7-4127-a4f4-124dbb68c405-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k\" (UID: \"51fcbefe-e2d7-4127-a4f4-124dbb68c405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:09.826821 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.826800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfck5\" (UniqueName: \"kubernetes.io/projected/51fcbefe-e2d7-4127-a4f4-124dbb68c405-kube-api-access-rfck5\") pod \"payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k\" (UID: \"51fcbefe-e2d7-4127-a4f4-124dbb68c405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:09.919338 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.919237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnp4l\" (UniqueName: \"kubernetes.io/projected/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-kube-api-access-gnp4l\") pod \"payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m\" (UID: \"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21\") " pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:09.919338 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.919311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m\" (UID: \"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21\") " pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:09.920080 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.920053 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-plugins-config-volume\") pod \"payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m\" (UID: \"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21\") " pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:09.926622 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.926597 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnp4l\" (UniqueName: \"kubernetes.io/projected/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-kube-api-access-gnp4l\") pod \"payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m\" (UID: \"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21\") " pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:09.959478 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:09.959449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:10.001543 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:10.001514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:10.106456 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:10.106163 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k"] Jul 30 11:19:10.109465 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:19:10.109438 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fcbefe_e2d7_4127_a4f4_124dbb68c405.slice/crio-0deade7acce22ac8d27da3dd783574831e45e02e5e1219d1a205ecc9937c2de2 WatchSource:0}: Error finding container 0deade7acce22ac8d27da3dd783574831e45e02e5e1219d1a205ecc9937c2de2: Status 404 returned error can't find the container with id 0deade7acce22ac8d27da3dd783574831e45e02e5e1219d1a205ecc9937c2de2 Jul 30 11:19:10.132970 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:10.132934 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" event={"ID":"51fcbefe-e2d7-4127-a4f4-124dbb68c405","Type":"ContainerStarted","Data":"0deade7acce22ac8d27da3dd783574831e45e02e5e1219d1a205ecc9937c2de2"} Jul 30 11:19:10.135676 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:10.135614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" event={"ID":"3617fe7f-0f1c-494b-8edb-9648ab127547","Type":"ContainerStarted","Data":"7bb53cba9f8a96703a7ba4000b8acf7f9f4f42465e7b9c4753efde3277dcf4b4"} Jul 30 11:19:10.161586 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:10.161563 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m"] Jul 30 11:19:10.162123 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:19:10.162073 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e15e2b_cf54_4e6a_a141_4f8b5e69cd21.slice/crio-8393e55366b1920c2ef9a48a7c7cb6afb634c29093fcdf8cdabf6e1e4bfa2e94 WatchSource:0}: Error finding container 8393e55366b1920c2ef9a48a7c7cb6afb634c29093fcdf8cdabf6e1e4bfa2e94: Status 404 returned error can't find the container with id 8393e55366b1920c2ef9a48a7c7cb6afb634c29093fcdf8cdabf6e1e4bfa2e94 Jul 30 11:19:11.147563 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:11.147512 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" event={"ID":"51fcbefe-e2d7-4127-a4f4-124dbb68c405","Type":"ContainerStarted","Data":"238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386"} Jul 30 11:19:11.148065 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:11.147724 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:11.149371 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:11.149350 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" event={"ID":"3617fe7f-0f1c-494b-8edb-9648ab127547","Type":"ContainerStarted","Data":"ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955"} Jul 30 11:19:11.149481 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:11.149403 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:11.151124 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:11.151078 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" event={"ID":"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21","Type":"ContainerStarted","Data":"0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26"} Jul 30 11:19:11.151249 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:11.151143 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" event={"ID":"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21","Type":"ContainerStarted","Data":"8393e55366b1920c2ef9a48a7c7cb6afb634c29093fcdf8cdabf6e1e4bfa2e94"} Jul 30 11:19:11.151249 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:11.151162 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:11.164933 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:11.164884 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" podStartSLOduration=2.164868785 podStartE2EDuration="2.164868785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:19:11.162891732 +0000 UTC m=+1859.135752796" watchObservedRunningTime="2026-07-30 11:19:11.164868785 +0000 UTC m=+1859.137729798" Jul 30 11:19:11.179475 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:11.179430 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" podStartSLOduration=2.486546861 podStartE2EDuration="3.17941627s" totalImagesPullingTime="692.869409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:19:11.177954508 +0000 UTC m=+1859.150815543" watchObservedRunningTime="2026-07-30 11:19:11.17941627 +0000 UTC m=+1859.152277305" Jul 30 11:19:11.194230 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:11.194182 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" podStartSLOduration=2.194171032 podStartE2EDuration="2.194171032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:19:11.191920251 +0000 UTC m=+1859.164781285" watchObservedRunningTime="2026-07-30 11:19:11.194171032 +0000 UTC m=+1859.167032067" Jul 30 11:19:17.163115 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:17.163056 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:22.159956 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:22.159916 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:22.160588 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:22.160258 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:39.502562 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:39.502473 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw"] Jul 30 11:19:39.502985 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:39.502747 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" podUID="3617fe7f-0f1c-494b-8edb-9648ab127547" containerName="maas-api" containerID="cri-o://ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955" gracePeriod=30 Jul 30 11:19:39.552256 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:39.552222 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m"] Jul 30 11:19:39.552497 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:39.552469 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" podUID="09e15e2b-cf54-4e6a-a141-4f8b5e69cd21" containerName="payload-processing" containerID="cri-o://0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26" gracePeriod=30 Jul 30 11:19:39.558808 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:39.558774 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k"] Jul 30 11:19:39.559259 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:39.559230 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" podUID="51fcbefe-e2d7-4127-a4f4-124dbb68c405" containerName="payload-pre-processing" containerID="cri-o://238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386" gracePeriod=30 Jul 30 11:19:39.838506 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:39.838071 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:39.868787 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:39.868764 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:39.871847 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:39.871830 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:40.011421 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.011330 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnp4l\" (UniqueName: \"kubernetes.io/projected/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-kube-api-access-gnp4l\") pod \"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21\" (UID: \"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21\") " Jul 30 11:19:40.011421 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.011385 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3617fe7f-0f1c-494b-8edb-9648ab127547-openshift-service-ca\") pod \"3617fe7f-0f1c-494b-8edb-9648ab127547\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " Jul 30 11:19:40.011649 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.011440 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twcjn\" (UniqueName: \"kubernetes.io/projected/3617fe7f-0f1c-494b-8edb-9648ab127547-kube-api-access-twcjn\") pod \"3617fe7f-0f1c-494b-8edb-9648ab127547\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " Jul 30 11:19:40.011649 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.011480 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfck5\" (UniqueName: \"kubernetes.io/projected/51fcbefe-e2d7-4127-a4f4-124dbb68c405-kube-api-access-rfck5\") pod \"51fcbefe-e2d7-4127-a4f4-124dbb68c405\" (UID: \"51fcbefe-e2d7-4127-a4f4-124dbb68c405\") " Jul 30 11:19:40.011649 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.011575 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fcbefe-e2d7-4127-a4f4-124dbb68c405-plugins-config-volume\") pod \"51fcbefe-e2d7-4127-a4f4-124dbb68c405\" (UID: \"51fcbefe-e2d7-4127-a4f4-124dbb68c405\") " Jul 30 11:19:40.011649 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.011599 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-plugins-config-volume\") pod \"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21\" (UID: \"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21\") " Jul 30 11:19:40.011649 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.011628 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3617fe7f-0f1c-494b-8edb-9648ab127547-maas-api-tls\") pod \"3617fe7f-0f1c-494b-8edb-9648ab127547\" (UID: \"3617fe7f-0f1c-494b-8edb-9648ab127547\") " Jul 30 11:19:40.011891 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.011831 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3617fe7f-0f1c-494b-8edb-9648ab127547-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3617fe7f-0f1c-494b-8edb-9648ab127547" (UID: "3617fe7f-0f1c-494b-8edb-9648ab127547"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:19:40.011951 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.011916 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fcbefe-e2d7-4127-a4f4-124dbb68c405-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51fcbefe-e2d7-4127-a4f4-124dbb68c405" (UID: "51fcbefe-e2d7-4127-a4f4-124dbb68c405"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:19:40.012034 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.012016 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51fcbefe-e2d7-4127-a4f4-124dbb68c405-plugins-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:19:40.012123 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.012041 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3617fe7f-0f1c-494b-8edb-9648ab127547-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:19:40.012340 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.012313 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09e15e2b-cf54-4e6a-a141-4f8b5e69cd21" (UID: "09e15e2b-cf54-4e6a-a141-4f8b5e69cd21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:19:40.013613 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.013589 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fcbefe-e2d7-4127-a4f4-124dbb68c405-kube-api-access-rfck5" (OuterVolumeSpecName: "kube-api-access-rfck5") pod "51fcbefe-e2d7-4127-a4f4-124dbb68c405" (UID: "51fcbefe-e2d7-4127-a4f4-124dbb68c405"). InnerVolumeSpecName "kube-api-access-rfck5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:19:40.013704 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.013663 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3617fe7f-0f1c-494b-8edb-9648ab127547-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3617fe7f-0f1c-494b-8edb-9648ab127547" (UID: "3617fe7f-0f1c-494b-8edb-9648ab127547"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 11:19:40.013749 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.013714 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-kube-api-access-gnp4l" (OuterVolumeSpecName: "kube-api-access-gnp4l") pod "09e15e2b-cf54-4e6a-a141-4f8b5e69cd21" (UID: "09e15e2b-cf54-4e6a-a141-4f8b5e69cd21"). InnerVolumeSpecName "kube-api-access-gnp4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:19:40.013749 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.013742 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3617fe7f-0f1c-494b-8edb-9648ab127547-kube-api-access-twcjn" (OuterVolumeSpecName: "kube-api-access-twcjn") pod "3617fe7f-0f1c-494b-8edb-9648ab127547" (UID: "3617fe7f-0f1c-494b-8edb-9648ab127547"). InnerVolumeSpecName "kube-api-access-twcjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:19:40.113081 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.113051 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnp4l\" (UniqueName: \"kubernetes.io/projected/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-kube-api-access-gnp4l\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:19:40.113081 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.113075 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twcjn\" (UniqueName: \"kubernetes.io/projected/3617fe7f-0f1c-494b-8edb-9648ab127547-kube-api-access-twcjn\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:19:40.113081 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.113105 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfck5\" (UniqueName: \"kubernetes.io/projected/51fcbefe-e2d7-4127-a4f4-124dbb68c405-kube-api-access-rfck5\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:19:40.113360 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.113116 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21-plugins-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:19:40.113360 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.113126 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3617fe7f-0f1c-494b-8edb-9648ab127547-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:19:40.325135 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.325082 2575 generic.go:356] "Generic (PLEG): container finished" podID="51fcbefe-e2d7-4127-a4f4-124dbb68c405" containerID="238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386" exitCode=0 Jul 30 11:19:40.325431 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.325225 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" event={"ID":"51fcbefe-e2d7-4127-a4f4-124dbb68c405","Type":"ContainerDied","Data":"238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386"} Jul 30 11:19:40.325431 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.325311 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" event={"ID":"51fcbefe-e2d7-4127-a4f4-124dbb68c405","Type":"ContainerDied","Data":"0deade7acce22ac8d27da3dd783574831e45e02e5e1219d1a205ecc9937c2de2"} Jul 30 11:19:40.325431 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.325346 2575 scope.go:117] "RemoveContainer" containerID="238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386" Jul 30 11:19:40.325627 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.325518 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k" Jul 30 11:19:40.328064 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.327598 2575 generic.go:356] "Generic (PLEG): container finished" podID="3617fe7f-0f1c-494b-8edb-9648ab127547" containerID="ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955" exitCode=0 Jul 30 11:19:40.328064 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.327647 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" Jul 30 11:19:40.328064 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.327689 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" event={"ID":"3617fe7f-0f1c-494b-8edb-9648ab127547","Type":"ContainerDied","Data":"ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955"} Jul 30 11:19:40.328064 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.327734 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw" event={"ID":"3617fe7f-0f1c-494b-8edb-9648ab127547","Type":"ContainerDied","Data":"7bb53cba9f8a96703a7ba4000b8acf7f9f4f42465e7b9c4753efde3277dcf4b4"} Jul 30 11:19:40.329465 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.329440 2575 generic.go:356] "Generic (PLEG): container finished" podID="09e15e2b-cf54-4e6a-a141-4f8b5e69cd21" containerID="0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26" exitCode=0 Jul 30 11:19:40.329574 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.329489 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" Jul 30 11:19:40.329574 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.329498 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" event={"ID":"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21","Type":"ContainerDied","Data":"0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26"} Jul 30 11:19:40.329574 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.329537 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m" event={"ID":"09e15e2b-cf54-4e6a-a141-4f8b5e69cd21","Type":"ContainerDied","Data":"8393e55366b1920c2ef9a48a7c7cb6afb634c29093fcdf8cdabf6e1e4bfa2e94"} Jul 30 11:19:40.337783 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.337410 2575 scope.go:117] "RemoveContainer" containerID="238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386" Jul 30 11:19:40.339989 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:19:40.339922 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386\": container with ID starting with 238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386 not found: ID does not exist" containerID="238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386" Jul 30 11:19:40.340130 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.340007 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386"} err="failed to get container status \"238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386\": rpc error: code = NotFound desc = could not find container \"238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386\": container with ID starting with 238dc55f8a6dfd23585eaa2da0c007b3cbe817dca33873664388b7202d4dc386 not found: ID does not exist" Jul 30 11:19:40.340772 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.340750 2575 scope.go:117] "RemoveContainer" containerID="ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955" Jul 30 11:19:40.354406 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.354384 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k"] Jul 30 11:19:40.355265 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.355250 2575 scope.go:117] "RemoveContainer" containerID="ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955" Jul 30 11:19:40.355538 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:19:40.355520 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955\": container with ID starting with ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955 not found: ID does not exist" containerID="ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955" Jul 30 11:19:40.355586 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.355547 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955"} err="failed to get container status \"ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955\": rpc error: code = NotFound desc = could not find container \"ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955\": container with ID starting with ca8af6a1ce3fb2643e220989dcfd85e0729ded9700abeeb6d1937dc04c20b955 not found: ID does not exist" Jul 30 11:19:40.355586 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.355564 2575 scope.go:117] "RemoveContainer" containerID="0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26" Jul 30 11:19:40.361649 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.361631 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2ca37a8f-6fb4985fb-5vf2k"] Jul 30 11:19:40.364297 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.364281 2575 scope.go:117] "RemoveContainer" containerID="0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26" Jul 30 11:19:40.364546 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:19:40.364525 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26\": container with ID starting with 0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26 not found: ID does not exist" containerID="0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26" Jul 30 11:19:40.364606 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.364553 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26"} err="failed to get container status \"0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26\": rpc error: code = NotFound desc = could not find container \"0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26\": container with ID starting with 0707e4333dcd69d3105a1cf71e167f5376e412a1b1895b92d561b5740c18bb26 not found: ID does not exist" Jul 30 11:19:40.370703 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.370679 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw"] Jul 30 11:19:40.377330 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.377310 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2ca37a8f-57488f4f54-kmrxw"] Jul 30 11:19:40.386065 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.386046 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m"] Jul 30 11:19:40.390518 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.390496 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2ca37a8f-697bdbcf6b-xh47m"] Jul 30 11:19:40.527885 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.527859 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e15e2b-cf54-4e6a-a141-4f8b5e69cd21" path="/var/lib/kubelet/pods/09e15e2b-cf54-4e6a-a141-4f8b5e69cd21/volumes" Jul 30 11:19:40.528320 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.528304 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3617fe7f-0f1c-494b-8edb-9648ab127547" path="/var/lib/kubelet/pods/3617fe7f-0f1c-494b-8edb-9648ab127547/volumes" Jul 30 11:19:40.528664 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.528651 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51fcbefe-e2d7-4127-a4f4-124dbb68c405" path="/var/lib/kubelet/pods/51fcbefe-e2d7-4127-a4f4-124dbb68c405/volumes" Jul 30 11:19:40.846236 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.846202 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl"] Jul 30 11:19:40.846713 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.846686 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fcbefe-e2d7-4127-a4f4-124dbb68c405" containerName="payload-pre-processing" Jul 30 11:19:40.846713 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.846707 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fcbefe-e2d7-4127-a4f4-124dbb68c405" containerName="payload-pre-processing" Jul 30 11:19:40.846936 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.846725 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3617fe7f-0f1c-494b-8edb-9648ab127547" containerName="maas-api" Jul 30 11:19:40.846936 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.846734 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3617fe7f-0f1c-494b-8edb-9648ab127547" containerName="maas-api" Jul 30 11:19:40.846936 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.846754 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e15e2b-cf54-4e6a-a141-4f8b5e69cd21" containerName="payload-processing" Jul 30 11:19:40.846936 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.846763 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e15e2b-cf54-4e6a-a141-4f8b5e69cd21" containerName="payload-processing" Jul 30 11:19:40.846936 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.846877 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3617fe7f-0f1c-494b-8edb-9648ab127547" containerName="maas-api" Jul 30 11:19:40.846936 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.846891 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e15e2b-cf54-4e6a-a141-4f8b5e69cd21" containerName="payload-processing" Jul 30 11:19:40.846936 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.846901 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fcbefe-e2d7-4127-a4f4-124dbb68c405" containerName="payload-pre-processing" Jul 30 11:19:40.851984 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.851968 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:40.853897 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.853877 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d9108b46-gw-openshift-default-dockercfg-s5tst\"" Jul 30 11:19:40.862930 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:40.862907 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl"] Jul 30 11:19:41.022377 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.022337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-credential-socket\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.022554 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.022419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-data\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.022554 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.022462 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-token\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.022554 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.022489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-certs\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.022554 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.022521 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-envoy\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.022797 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.022579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-socket\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.022797 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.022613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-podinfo\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.022797 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.022647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istiod-ca-cert\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.022797 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.022691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6924\" (UniqueName: \"kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-kube-api-access-f6924\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124141 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-credential-socket\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124141 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-data\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124141 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-token\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124436 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124163 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-certs\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124436 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-envoy\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124436 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124350 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-socket\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124436 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-podinfo\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124436 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istiod-ca-cert\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124671 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-credential-socket\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124671 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6924\" (UniqueName: \"kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-kube-api-access-f6924\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124671 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124497 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-certs\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124671 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-data\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.124831 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.124755 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-socket\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.125022 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.125002 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istiod-ca-cert\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.126518 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.126497 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-envoy\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.126820 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.126803 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-podinfo\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.131352 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.131331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-token\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.131490 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.131472 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6924\" (UniqueName: \"kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-kube-api-access-f6924\") pod \"e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.164641 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.164612 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:41.302597 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.302572 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl"] Jul 30 11:19:41.303904 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:19:41.303879 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbf3d10_7442_44d8_87e3_9447b9f820aa.slice/crio-aff560f5ab64a6e4a80bb9f7942344953f1ea5ec717795e92835ae5eb1bb6032 WatchSource:0}: Error finding container aff560f5ab64a6e4a80bb9f7942344953f1ea5ec717795e92835ae5eb1bb6032: Status 404 returned error can't find the container with id aff560f5ab64a6e4a80bb9f7942344953f1ea5ec717795e92835ae5eb1bb6032 Jul 30 11:19:41.338449 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:41.338423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" event={"ID":"8dbf3d10-7442-44d8-87e3-9447b9f820aa","Type":"ContainerStarted","Data":"aff560f5ab64a6e4a80bb9f7942344953f1ea5ec717795e92835ae5eb1bb6032"} Jul 30 11:19:46.354429 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:46.354397 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:19:46.354703 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:46.354469 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:19:46.354703 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:46.354494 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:19:47.380043 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:47.380013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" event={"ID":"8dbf3d10-7442-44d8-87e3-9447b9f820aa","Type":"ContainerStarted","Data":"2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3"} Jul 30 11:19:47.400283 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:47.400230 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" podStartSLOduration=2.352109561 podStartE2EDuration="7.400215052s" totalImagesPullingTime="5.048105491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:19:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:19:47.397513227 +0000 UTC m=+1895.370374274" watchObservedRunningTime="2026-07-30 11:19:47.400215052 +0000 UTC m=+1895.373076087" Jul 30 11:19:48.164944 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:48.164904 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:48.170071 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:48.170046 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:48.386407 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:48.386377 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:48.387278 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:48.387262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:19:48.983109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:48.983062 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8"] Jul 30 11:19:48.987271 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:48.987247 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:48.989201 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:48.989173 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d9108b46-serving-cert\"" Jul 30 11:19:48.997948 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:48.997921 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8"] Jul 30 11:19:49.003265 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.003240 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-openshift-service-ca\") pod \"maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.003367 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.003297 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzm7q\" (UniqueName: \"kubernetes.io/projected/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-kube-api-access-mzm7q\") pod \"maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.003415 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.003392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-maas-api-tls\") pod \"maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.104174 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.104140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzm7q\" (UniqueName: \"kubernetes.io/projected/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-kube-api-access-mzm7q\") pod \"maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.104336 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.104187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-maas-api-tls\") pod \"maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.104336 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.104241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-openshift-service-ca\") pod \"maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.104424 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:19:49.104359 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-serving-cert: secret "maas-api-e2e-migrate-d9108b46-serving-cert" not found Jul 30 11:19:49.104460 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:19:49.104436 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-maas-api-tls podName:0a3656f8-d1e4-4c11-ac41-544bac7dd8af nodeName:}" failed. No retries permitted until 2026-07-30 11:19:49.604420322 +0000 UTC m=+1897.577281336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-maas-api-tls") pod "maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" (UID: "0a3656f8-d1e4-4c11-ac41-544bac7dd8af") : secret "maas-api-e2e-migrate-d9108b46-serving-cert" not found Jul 30 11:19:49.104800 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.104784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-openshift-service-ca\") pod \"maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.114683 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.114655 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzm7q\" (UniqueName: \"kubernetes.io/projected/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-kube-api-access-mzm7q\") pod \"maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.607355 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.607321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-maas-api-tls\") pod \"maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.609889 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.609862 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-maas-api-tls\") pod \"maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.619037 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.619009 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl"] Jul 30 11:19:49.623328 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.623304 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:19:49.626016 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.625991 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d9108b46\"" Jul 30 11:19:49.626426 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.626065 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d9108b46-dockercfg-p5gb9\"" Jul 30 11:19:49.635333 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.635307 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl"] Jul 30 11:19:49.707945 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.707915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/503994e3-beb9-485e-8c53-02f2e498c858-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl\" (UID: \"503994e3-beb9-485e-8c53-02f2e498c858\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:19:49.707945 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.707949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8gr6\" (UniqueName: \"kubernetes.io/projected/503994e3-beb9-485e-8c53-02f2e498c858-kube-api-access-s8gr6\") pod \"payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl\" (UID: \"503994e3-beb9-485e-8c53-02f2e498c858\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:19:49.809130 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.809068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/503994e3-beb9-485e-8c53-02f2e498c858-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl\" (UID: \"503994e3-beb9-485e-8c53-02f2e498c858\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:19:49.809334 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.809151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8gr6\" (UniqueName: \"kubernetes.io/projected/503994e3-beb9-485e-8c53-02f2e498c858-kube-api-access-s8gr6\") pod \"payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl\" (UID: \"503994e3-beb9-485e-8c53-02f2e498c858\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:19:49.809977 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.809952 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/503994e3-beb9-485e-8c53-02f2e498c858-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl\" (UID: \"503994e3-beb9-485e-8c53-02f2e498c858\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:19:49.817535 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.817511 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8gr6\" (UniqueName: \"kubernetes.io/projected/503994e3-beb9-485e-8c53-02f2e498c858-kube-api-access-s8gr6\") pod \"payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl\" (UID: \"503994e3-beb9-485e-8c53-02f2e498c858\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:19:49.900759 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.900677 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:49.943642 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:49.943608 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:19:50.084050 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:50.083970 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8"] Jul 30 11:19:50.085209 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:19:50.085182 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3656f8_d1e4_4c11_ac41_544bac7dd8af.slice/crio-0bb9bad6de048b111aa4735d0add8d02a3f8641fa1bd7aaf0a0654418f3c0cfb WatchSource:0}: Error finding container 0bb9bad6de048b111aa4735d0add8d02a3f8641fa1bd7aaf0a0654418f3c0cfb: Status 404 returned error can't find the container with id 0bb9bad6de048b111aa4735d0add8d02a3f8641fa1bd7aaf0a0654418f3c0cfb Jul 30 11:19:50.116587 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:50.116561 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl"] Jul 30 11:19:50.117020 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:19:50.116999 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503994e3_beb9_485e_8c53_02f2e498c858.slice/crio-e37290eea44ed66c72f69c0e0fc38afe8d020dce6f89e4bdc20ec69cd9c0a474 WatchSource:0}: Error finding container e37290eea44ed66c72f69c0e0fc38afe8d020dce6f89e4bdc20ec69cd9c0a474: Status 404 returned error can't find the container with id e37290eea44ed66c72f69c0e0fc38afe8d020dce6f89e4bdc20ec69cd9c0a474 Jul 30 11:19:50.400671 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:50.400631 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" event={"ID":"0a3656f8-d1e4-4c11-ac41-544bac7dd8af","Type":"ContainerStarted","Data":"0bb9bad6de048b111aa4735d0add8d02a3f8641fa1bd7aaf0a0654418f3c0cfb"} Jul 30 11:19:50.402064 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:50.402035 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" event={"ID":"503994e3-beb9-485e-8c53-02f2e498c858","Type":"ContainerStarted","Data":"bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c"} Jul 30 11:19:50.402197 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:50.402099 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" event={"ID":"503994e3-beb9-485e-8c53-02f2e498c858","Type":"ContainerStarted","Data":"e37290eea44ed66c72f69c0e0fc38afe8d020dce6f89e4bdc20ec69cd9c0a474"} Jul 30 11:19:50.402264 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:50.402253 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:19:50.419280 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:50.419188 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" podStartSLOduration=1.419173893 podStartE2EDuration="1.419173893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:19:50.417465838 +0000 UTC m=+1898.390326892" watchObservedRunningTime="2026-07-30 11:19:50.419173893 +0000 UTC m=+1898.392034927" Jul 30 11:19:51.409277 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:51.409237 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" event={"ID":"0a3656f8-d1e4-4c11-ac41-544bac7dd8af","Type":"ContainerStarted","Data":"9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69"} Jul 30 11:19:51.409685 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:51.409481 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:19:51.426663 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:51.426616 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" podStartSLOduration=3.071532749 podStartE2EDuration="3.426601592s" totalImagesPullingTime="355.068843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:19:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:19:51.42579524 +0000 UTC m=+1899.398656275" watchObservedRunningTime="2026-07-30 11:19:51.426601592 +0000 UTC m=+1899.399462627" Jul 30 11:19:57.421898 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:19:57.421865 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:20:01.412131 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:01.412078 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:20:02.493108 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:02.493054 2575 generic.go:356] "Generic (PLEG): container finished" podID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerID="2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3" exitCode=0 Jul 30 11:20:02.493612 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:02.493133 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" event={"ID":"8dbf3d10-7442-44d8-87e3-9447b9f820aa","Type":"ContainerDied","Data":"2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3"} Jul 30 11:20:02.493676 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:02.493647 2575 scope.go:117] "RemoveContainer" containerID="2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3" Jul 30 11:20:02.495692 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:02.495663 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:20:02.495793 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:02.495726 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:20:02.495793 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:02.495759 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:20:03.386573 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:03.386538 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:20:03.501481 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:03.501440 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" event={"ID":"8dbf3d10-7442-44d8-87e3-9447b9f820aa","Type":"ContainerStarted","Data":"0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e"} Jul 30 11:20:04.165119 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:04.165062 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:20:04.166327 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:04.166304 2575 patch_prober.go:28] interesting pod/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 30 11:20:04.166382 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:04.166352 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 30 11:20:05.165768 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:05.165732 2575 patch_prober.go:28] interesting pod/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 30 11:20:05.166168 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:05.165809 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 30 11:20:06.165342 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:06.165303 2575 patch_prober.go:28] interesting pod/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 30 11:20:06.165535 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:06.165366 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 30 11:20:07.165934 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:07.165894 2575 patch_prober.go:28] interesting pod/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 30 11:20:07.166441 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:07.165968 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 30 11:20:08.165989 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:08.165949 2575 patch_prober.go:28] interesting pod/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 30 11:20:08.166453 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:08.166021 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 30 11:20:09.169334 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:09.169298 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:20:09.169759 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:09.169545 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:20:09.170256 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:09.170236 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:20:20.512058 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.512022 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8"] Jul 30 11:20:20.512609 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.512319 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" podUID="0a3656f8-d1e4-4c11-ac41-544bac7dd8af" containerName="maas-api" containerID="cri-o://9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69" gracePeriod=30 Jul 30 11:20:20.562354 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.562318 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl"] Jul 30 11:20:20.562588 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.562537 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" podUID="503994e3-beb9-485e-8c53-02f2e498c858" containerName="payload-processing" containerID="cri-o://bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c" gracePeriod=30 Jul 30 11:20:20.830820 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.830784 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:20:20.860684 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.860379 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl"] Jul 30 11:20:20.860821 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.860670 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" containerID="cri-o://0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e" gracePeriod=30 Jul 30 11:20:20.869503 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.869472 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:20:20.910556 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.910523 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzm7q\" (UniqueName: \"kubernetes.io/projected/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-kube-api-access-mzm7q\") pod \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " Jul 30 11:20:20.910730 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.910572 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-maas-api-tls\") pod \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " Jul 30 11:20:20.910730 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.910672 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-openshift-service-ca\") pod \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\" (UID: \"0a3656f8-d1e4-4c11-ac41-544bac7dd8af\") " Jul 30 11:20:20.911038 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.911015 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a3656f8-d1e4-4c11-ac41-544bac7dd8af" (UID: "0a3656f8-d1e4-4c11-ac41-544bac7dd8af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:20:20.912588 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.912556 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a3656f8-d1e4-4c11-ac41-544bac7dd8af" (UID: "0a3656f8-d1e4-4c11-ac41-544bac7dd8af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 11:20:20.912588 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:20.912582 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-kube-api-access-mzm7q" (OuterVolumeSpecName: "kube-api-access-mzm7q") pod "0a3656f8-d1e4-4c11-ac41-544bac7dd8af" (UID: "0a3656f8-d1e4-4c11-ac41-544bac7dd8af"). InnerVolumeSpecName "kube-api-access-mzm7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:20:21.011663 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.011631 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/503994e3-beb9-485e-8c53-02f2e498c858-plugins-config-volume\") pod \"503994e3-beb9-485e-8c53-02f2e498c858\" (UID: \"503994e3-beb9-485e-8c53-02f2e498c858\") " Jul 30 11:20:21.011829 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.011673 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8gr6\" (UniqueName: \"kubernetes.io/projected/503994e3-beb9-485e-8c53-02f2e498c858-kube-api-access-s8gr6\") pod \"503994e3-beb9-485e-8c53-02f2e498c858\" (UID: \"503994e3-beb9-485e-8c53-02f2e498c858\") " Jul 30 11:20:21.011890 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.011843 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:21.011890 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.011854 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzm7q\" (UniqueName: \"kubernetes.io/projected/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-kube-api-access-mzm7q\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:21.011890 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.011864 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a3656f8-d1e4-4c11-ac41-544bac7dd8af-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:21.012073 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.012049 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503994e3-beb9-485e-8c53-02f2e498c858-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "503994e3-beb9-485e-8c53-02f2e498c858" (UID: "503994e3-beb9-485e-8c53-02f2e498c858"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:20:21.013760 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.013740 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503994e3-beb9-485e-8c53-02f2e498c858-kube-api-access-s8gr6" (OuterVolumeSpecName: "kube-api-access-s8gr6") pod "503994e3-beb9-485e-8c53-02f2e498c858" (UID: "503994e3-beb9-485e-8c53-02f2e498c858"). InnerVolumeSpecName "kube-api-access-s8gr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:20:21.112927 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.112893 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/503994e3-beb9-485e-8c53-02f2e498c858-plugins-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:21.112927 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.112919 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8gr6\" (UniqueName: \"kubernetes.io/projected/503994e3-beb9-485e-8c53-02f2e498c858-kube-api-access-s8gr6\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:21.609588 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.609552 2575 generic.go:356] "Generic (PLEG): container finished" podID="0a3656f8-d1e4-4c11-ac41-544bac7dd8af" containerID="9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69" exitCode=0 Jul 30 11:20:21.610046 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.609617 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" Jul 30 11:20:21.610046 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.609644 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" event={"ID":"0a3656f8-d1e4-4c11-ac41-544bac7dd8af","Type":"ContainerDied","Data":"9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69"} Jul 30 11:20:21.610046 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.609715 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8" event={"ID":"0a3656f8-d1e4-4c11-ac41-544bac7dd8af","Type":"ContainerDied","Data":"0bb9bad6de048b111aa4735d0add8d02a3f8641fa1bd7aaf0a0654418f3c0cfb"} Jul 30 11:20:21.610046 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.609749 2575 scope.go:117] "RemoveContainer" containerID="9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69" Jul 30 11:20:21.611528 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.611499 2575 generic.go:356] "Generic (PLEG): container finished" podID="503994e3-beb9-485e-8c53-02f2e498c858" containerID="bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c" exitCode=0 Jul 30 11:20:21.611633 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.611552 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" Jul 30 11:20:21.611633 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.611555 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" event={"ID":"503994e3-beb9-485e-8c53-02f2e498c858","Type":"ContainerDied","Data":"bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c"} Jul 30 11:20:21.611633 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.611620 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl" event={"ID":"503994e3-beb9-485e-8c53-02f2e498c858","Type":"ContainerDied","Data":"e37290eea44ed66c72f69c0e0fc38afe8d020dce6f89e4bdc20ec69cd9c0a474"} Jul 30 11:20:21.621851 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.621831 2575 scope.go:117] "RemoveContainer" containerID="9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69" Jul 30 11:20:21.622226 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:20:21.622201 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69\": container with ID starting with 9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69 not found: ID does not exist" containerID="9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69" Jul 30 11:20:21.622315 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.622238 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69"} err="failed to get container status \"9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69\": rpc error: code = NotFound desc = could not find container \"9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69\": container with ID starting with 9b5affb45fb3ea44fadaf1841c200e43554807aaf0620858e55c9c9e542fec69 not found: ID does not exist" Jul 30 11:20:21.622315 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.622259 2575 scope.go:117] "RemoveContainer" containerID="bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c" Jul 30 11:20:21.633192 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.633173 2575 scope.go:117] "RemoveContainer" containerID="bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c" Jul 30 11:20:21.633459 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:20:21.633441 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c\": container with ID starting with bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c not found: ID does not exist" containerID="bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c" Jul 30 11:20:21.633514 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.633467 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c"} err="failed to get container status \"bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c\": rpc error: code = NotFound desc = could not find container \"bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c\": container with ID starting with bafcbf4a3e4b35ca37f88a9905734259cafb2dc561580b9ad9dcdb63f4fcd55c not found: ID does not exist" Jul 30 11:20:21.643786 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.643759 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8"] Jul 30 11:20:21.650650 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.650627 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9108b46-84c575fcbd-7qqq8"] Jul 30 11:20:21.661496 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.661471 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl"] Jul 30 11:20:21.672021 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:21.671986 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9108b46-58b56fff57-vvwhl"] Jul 30 11:20:22.527436 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:22.527397 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a3656f8-d1e4-4c11-ac41-544bac7dd8af" path="/var/lib/kubelet/pods/0a3656f8-d1e4-4c11-ac41-544bac7dd8af/volumes" Jul 30 11:20:22.527842 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:22.527821 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="503994e3-beb9-485e-8c53-02f2e498c858" path="/var/lib/kubelet/pods/503994e3-beb9-485e-8c53-02f2e498c858/volumes" Jul 30 11:20:24.172465 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:24.172429 2575 patch_prober.go:28] interesting pod/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 11:20:24.172827 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:24.172487 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 11:20:26.155018 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.154992 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:20:26.215380 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.215291 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7"] Jul 30 11:20:26.215798 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.215781 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="503994e3-beb9-485e-8c53-02f2e498c858" containerName="payload-processing" Jul 30 11:20:26.215846 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.215800 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="503994e3-beb9-485e-8c53-02f2e498c858" containerName="payload-processing" Jul 30 11:20:26.215846 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.215815 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" Jul 30 11:20:26.215846 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.215820 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" Jul 30 11:20:26.215846 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.215839 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a3656f8-d1e4-4c11-ac41-544bac7dd8af" containerName="maas-api" Jul 30 11:20:26.215846 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.215845 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a3656f8-d1e4-4c11-ac41-544bac7dd8af" containerName="maas-api" Jul 30 11:20:26.216051 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.215909 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" Jul 30 11:20:26.216051 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.215921 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="503994e3-beb9-485e-8c53-02f2e498c858" containerName="payload-processing" Jul 30 11:20:26.216051 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.215929 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a3656f8-d1e4-4c11-ac41-544bac7dd8af" containerName="maas-api" Jul 30 11:20:26.218873 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.218856 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:26.220895 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.220877 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-519ed3f7\"" Jul 30 11:20:26.220977 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.220914 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-519ed3f7-dockercfg-7cqs4\"" Jul 30 11:20:26.230566 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.230478 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7"] Jul 30 11:20:26.266370 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.266330 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-token\") pod \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " Jul 30 11:20:26.266564 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.266439 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-socket\") pod \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " Jul 30 11:20:26.266564 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.266474 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-podinfo\") pod \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " Jul 30 11:20:26.266564 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.266507 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6924\" (UniqueName: \"kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-kube-api-access-f6924\") pod \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " Jul 30 11:20:26.266564 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.266533 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istiod-ca-cert\") pod \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " Jul 30 11:20:26.266808 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.266568 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-envoy\") pod \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " Jul 30 11:20:26.266808 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.266596 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-data\") pod \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " Jul 30 11:20:26.266808 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.266648 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-certs\") pod \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " Jul 30 11:20:26.266808 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.266690 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-credential-socket\") pod \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\" (UID: \"8dbf3d10-7442-44d8-87e3-9447b9f820aa\") " Jul 30 11:20:26.267554 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.267476 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8dbf3d10-7442-44d8-87e3-9447b9f820aa" (UID: "8dbf3d10-7442-44d8-87e3-9447b9f820aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:20:26.267733 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.267660 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8dbf3d10-7442-44d8-87e3-9447b9f820aa" (UID: "8dbf3d10-7442-44d8-87e3-9447b9f820aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:20:26.268429 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.268080 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8dbf3d10-7442-44d8-87e3-9447b9f820aa" (UID: "8dbf3d10-7442-44d8-87e3-9447b9f820aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:20:26.268552 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.268482 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8dbf3d10-7442-44d8-87e3-9447b9f820aa" (UID: "8dbf3d10-7442-44d8-87e3-9447b9f820aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:20:26.268617 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.268555 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "8dbf3d10-7442-44d8-87e3-9447b9f820aa" (UID: "8dbf3d10-7442-44d8-87e3-9447b9f820aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:20:26.269109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.269060 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "8dbf3d10-7442-44d8-87e3-9447b9f820aa" (UID: "8dbf3d10-7442-44d8-87e3-9447b9f820aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:20:26.270160 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.270137 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8dbf3d10-7442-44d8-87e3-9447b9f820aa" (UID: "8dbf3d10-7442-44d8-87e3-9447b9f820aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 11:20:26.270241 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.270151 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-kube-api-access-f6924" (OuterVolumeSpecName: "kube-api-access-f6924") pod "8dbf3d10-7442-44d8-87e3-9447b9f820aa" (UID: "8dbf3d10-7442-44d8-87e3-9447b9f820aa"). InnerVolumeSpecName "kube-api-access-f6924". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:20:26.270241 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.270160 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8dbf3d10-7442-44d8-87e3-9447b9f820aa" (UID: "8dbf3d10-7442-44d8-87e3-9447b9f820aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:20:26.367973 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.367942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lmfp\" (UniqueName: \"kubernetes.io/projected/0ab66459-3212-49b9-9910-ac308d3b5e73-kube-api-access-6lmfp\") pod \"payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7\" (UID: \"0ab66459-3212-49b9-9910-ac308d3b5e73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:26.368149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.367988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab66459-3212-49b9-9910-ac308d3b5e73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7\" (UID: \"0ab66459-3212-49b9-9910-ac308d3b5e73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:26.368149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.368074 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-socket\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:26.368149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.368097 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-podinfo\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:26.368149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.368109 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6924\" (UniqueName: \"kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-kube-api-access-f6924\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:26.368149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.368119 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istiod-ca-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:26.368149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.368130 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-envoy\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:26.368149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.368139 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-data\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:26.368149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.368147 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-workload-certs\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:26.368417 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.368155 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbf3d10-7442-44d8-87e3-9447b9f820aa-credential-socket\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:26.368417 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.368163 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbf3d10-7442-44d8-87e3-9447b9f820aa-istio-token\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:26.468791 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.468711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lmfp\" (UniqueName: \"kubernetes.io/projected/0ab66459-3212-49b9-9910-ac308d3b5e73-kube-api-access-6lmfp\") pod \"payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7\" (UID: \"0ab66459-3212-49b9-9910-ac308d3b5e73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:26.468791 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.468770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab66459-3212-49b9-9910-ac308d3b5e73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7\" (UID: \"0ab66459-3212-49b9-9910-ac308d3b5e73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:26.469465 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.469447 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab66459-3212-49b9-9910-ac308d3b5e73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7\" (UID: \"0ab66459-3212-49b9-9910-ac308d3b5e73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:26.476353 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.476325 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lmfp\" (UniqueName: \"kubernetes.io/projected/0ab66459-3212-49b9-9910-ac308d3b5e73-kube-api-access-6lmfp\") pod \"payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7\" (UID: \"0ab66459-3212-49b9-9910-ac308d3b5e73\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:26.531133 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.531106 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:26.648544 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.648491 2575 generic.go:356] "Generic (PLEG): container finished" podID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerID="0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e" exitCode=0 Jul 30 11:20:26.648544 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.648546 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" event={"ID":"8dbf3d10-7442-44d8-87e3-9447b9f820aa","Type":"ContainerDied","Data":"0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e"} Jul 30 11:20:26.648859 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.648561 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" Jul 30 11:20:26.648859 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.648600 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl" event={"ID":"8dbf3d10-7442-44d8-87e3-9447b9f820aa","Type":"ContainerDied","Data":"aff560f5ab64a6e4a80bb9f7942344953f1ea5ec717795e92835ae5eb1bb6032"} Jul 30 11:20:26.648859 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.648632 2575 scope.go:117] "RemoveContainer" containerID="0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e" Jul 30 11:20:26.658939 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.658920 2575 scope.go:117] "RemoveContainer" containerID="2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3" Jul 30 11:20:26.668008 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.667979 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7"] Jul 30 11:20:26.671521 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.671499 2575 scope.go:117] "RemoveContainer" containerID="0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e" Jul 30 11:20:26.671836 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:20:26.671817 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e\": container with ID starting with 0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e not found: ID does not exist" containerID="0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e" Jul 30 11:20:26.671917 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.671843 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e"} err="failed to get container status \"0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e\": rpc error: code = NotFound desc = could not find container \"0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e\": container with ID starting with 0a4a97039cf0375682c8d3defdffc83a6b249ef37cead6f102245703f4e5332e not found: ID does not exist" Jul 30 11:20:26.671917 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.671861 2575 scope.go:117] "RemoveContainer" containerID="2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3" Jul 30 11:20:26.672053 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.672038 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl"] Jul 30 11:20:26.672168 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:20:26.672137 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3\": container with ID starting with 2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3 not found: ID does not exist" containerID="2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3" Jul 30 11:20:26.672168 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.672154 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3"} err="failed to get container status \"2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3\": rpc error: code = NotFound desc = could not find container \"2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3\": container with ID starting with 2b37ac031afef2112c5a872bc234cbd13cc1b9be64b2ac281105ff899f8160f3 not found: ID does not exist" Jul 30 11:20:26.672719 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:20:26.672700 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab66459_3212_49b9_9910_ac308d3b5e73.slice/crio-869d8cc7e5ed88350f01497e6c12bda594ad1746e917bc03bdc2d2a9f2f9867e WatchSource:0}: Error finding container 869d8cc7e5ed88350f01497e6c12bda594ad1746e917bc03bdc2d2a9f2f9867e: Status 404 returned error can't find the container with id 869d8cc7e5ed88350f01497e6c12bda594ad1746e917bc03bdc2d2a9f2f9867e Jul 30 11:20:26.674625 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:26.674606 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d9108b46-gw-openshift-default-7df554fc6-z9mnl"] Jul 30 11:20:27.659959 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:27.659922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" event={"ID":"0ab66459-3212-49b9-9910-ac308d3b5e73","Type":"ContainerStarted","Data":"f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f"} Jul 30 11:20:27.660460 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:27.660007 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" event={"ID":"0ab66459-3212-49b9-9910-ac308d3b5e73","Type":"ContainerStarted","Data":"869d8cc7e5ed88350f01497e6c12bda594ad1746e917bc03bdc2d2a9f2f9867e"} Jul 30 11:20:27.660460 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:27.660081 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:27.676273 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:27.676224 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" podStartSLOduration=1.67620973 podStartE2EDuration="1.67620973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:20:27.675294991 +0000 UTC m=+1935.648156045" watchObservedRunningTime="2026-07-30 11:20:27.67620973 +0000 UTC m=+1935.649070765" Jul 30 11:20:28.528073 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:28.528033 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" path="/var/lib/kubelet/pods/8dbf3d10-7442-44d8-87e3-9447b9f820aa/volumes" Jul 30 11:20:38.668920 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:38.668886 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:55.988432 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:55.988385 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7"] Jul 30 11:20:55.989027 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:55.988657 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" podUID="0ab66459-3212-49b9-9910-ac308d3b5e73" containerName="payload-pre-processing" containerID="cri-o://f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f" gracePeriod=30 Jul 30 11:20:56.242828 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.242762 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:56.355006 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.354962 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lmfp\" (UniqueName: \"kubernetes.io/projected/0ab66459-3212-49b9-9910-ac308d3b5e73-kube-api-access-6lmfp\") pod \"0ab66459-3212-49b9-9910-ac308d3b5e73\" (UID: \"0ab66459-3212-49b9-9910-ac308d3b5e73\") " Jul 30 11:20:56.355006 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.355019 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab66459-3212-49b9-9910-ac308d3b5e73-plugins-config-volume\") pod \"0ab66459-3212-49b9-9910-ac308d3b5e73\" (UID: \"0ab66459-3212-49b9-9910-ac308d3b5e73\") " Jul 30 11:20:56.355438 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.355409 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab66459-3212-49b9-9910-ac308d3b5e73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ab66459-3212-49b9-9910-ac308d3b5e73" (UID: "0ab66459-3212-49b9-9910-ac308d3b5e73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:20:56.356994 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.356970 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab66459-3212-49b9-9910-ac308d3b5e73-kube-api-access-6lmfp" (OuterVolumeSpecName: "kube-api-access-6lmfp") pod "0ab66459-3212-49b9-9910-ac308d3b5e73" (UID: "0ab66459-3212-49b9-9910-ac308d3b5e73"). InnerVolumeSpecName "kube-api-access-6lmfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:20:56.456535 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.456506 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lmfp\" (UniqueName: \"kubernetes.io/projected/0ab66459-3212-49b9-9910-ac308d3b5e73-kube-api-access-6lmfp\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:56.456535 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.456531 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab66459-3212-49b9-9910-ac308d3b5e73-plugins-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:20:56.834727 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.834634 2575 generic.go:356] "Generic (PLEG): container finished" podID="0ab66459-3212-49b9-9910-ac308d3b5e73" containerID="f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f" exitCode=0 Jul 30 11:20:56.834727 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.834697 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" Jul 30 11:20:56.834727 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.834717 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" event={"ID":"0ab66459-3212-49b9-9910-ac308d3b5e73","Type":"ContainerDied","Data":"f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f"} Jul 30 11:20:56.834998 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.834765 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7" event={"ID":"0ab66459-3212-49b9-9910-ac308d3b5e73","Type":"ContainerDied","Data":"869d8cc7e5ed88350f01497e6c12bda594ad1746e917bc03bdc2d2a9f2f9867e"} Jul 30 11:20:56.834998 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.834788 2575 scope.go:117] "RemoveContainer" containerID="f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f" Jul 30 11:20:56.843801 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.843783 2575 scope.go:117] "RemoveContainer" containerID="f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f" Jul 30 11:20:56.844072 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:20:56.844053 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f\": container with ID starting with f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f not found: ID does not exist" containerID="f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f" Jul 30 11:20:56.844149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.844080 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f"} err="failed to get container status \"f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f\": rpc error: code = NotFound desc = could not find container \"f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f\": container with ID starting with f3ac187d4ac07c7fefbc41401776f1f3489094935c80935c494966bd4e96266f not found: ID does not exist" Jul 30 11:20:56.850223 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.850200 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7"] Jul 30 11:20:56.855754 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:56.855733 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519ed3f7-59c58ff96-spqk7"] Jul 30 11:20:58.021156 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.021126 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25"] Jul 30 11:20:58.021627 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.021570 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" Jul 30 11:20:58.021627 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.021581 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" Jul 30 11:20:58.021627 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.021604 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab66459-3212-49b9-9910-ac308d3b5e73" containerName="payload-pre-processing" Jul 30 11:20:58.021627 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.021610 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab66459-3212-49b9-9910-ac308d3b5e73" containerName="payload-pre-processing" Jul 30 11:20:58.021810 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.021698 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dbf3d10-7442-44d8-87e3-9447b9f820aa" containerName="istio-proxy" Jul 30 11:20:58.021810 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.021714 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab66459-3212-49b9-9910-ac308d3b5e73" containerName="payload-pre-processing" Jul 30 11:20:58.026298 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.026277 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.028299 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.028279 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-772a6e1a-openshift-default-dockercfg-lwl4t\"" Jul 30 11:20:58.036976 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.036952 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25"] Jul 30 11:20:58.070824 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.070794 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2jwc\" (UniqueName: \"kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-kube-api-access-k2jwc\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.070956 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.070827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-credential-socket\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.070956 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.070855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-envoy\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.070956 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.070934 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-certs\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.071061 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.070972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-socket\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.071061 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.071021 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istiod-ca-cert\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.071061 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.071055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-token\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.071182 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.071117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-podinfo\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.071182 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.071160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-data\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.171732 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.171702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-socket\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.171895 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.171751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istiod-ca-cert\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.171895 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.171874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-token\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.171980 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.171911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-podinfo\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.171980 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.171964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-data\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.172182 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.171989 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2jwc\" (UniqueName: \"kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-kube-api-access-k2jwc\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.172182 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.172017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-credential-socket\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.172182 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.172055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-envoy\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.172182 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.172078 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-socket\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.172182 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.172142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-certs\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.172473 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.172312 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-data\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.172473 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.172372 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-credential-socket\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.172583 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.172511 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istiod-ca-cert\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.172583 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.172521 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-certs\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.174374 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.174352 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-envoy\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.174536 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.174518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-podinfo\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.179562 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.179542 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-token\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.179673 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.179546 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2jwc\" (UniqueName: \"kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-kube-api-access-k2jwc\") pod \"e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.337748 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.337716 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:58.479556 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.479525 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25"] Jul 30 11:20:58.481013 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:20:58.480990 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f10a3dc_ff94_4d2b_be0e_e6947e4cbc19.slice/crio-146c8852dac6aced82aa11d04b83e01da9ca8232974a4d0ba6f7b3b5f5d10b05 WatchSource:0}: Error finding container 146c8852dac6aced82aa11d04b83e01da9ca8232974a4d0ba6f7b3b5f5d10b05: Status 404 returned error can't find the container with id 146c8852dac6aced82aa11d04b83e01da9ca8232974a4d0ba6f7b3b5f5d10b05 Jul 30 11:20:58.483605 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.483576 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:20:58.483726 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.483638 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:20:58.483726 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.483667 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:20:58.529942 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.529913 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab66459-3212-49b9-9910-ac308d3b5e73" path="/var/lib/kubelet/pods/0ab66459-3212-49b9-9910-ac308d3b5e73/volumes" Jul 30 11:20:58.852447 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.852411 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" event={"ID":"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19","Type":"ContainerStarted","Data":"a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54"} Jul 30 11:20:58.852649 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.852469 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" event={"ID":"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19","Type":"ContainerStarted","Data":"146c8852dac6aced82aa11d04b83e01da9ca8232974a4d0ba6f7b3b5f5d10b05"} Jul 30 11:20:58.873077 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:58.873030 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" podStartSLOduration=0.873014613 podStartE2EDuration="873.014613ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:20:58.869982018 +0000 UTC m=+1966.842843053" watchObservedRunningTime="2026-07-30 11:20:58.873014613 +0000 UTC m=+1966.845875647" Jul 30 11:20:59.338688 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:59.338651 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:59.345611 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:59.345580 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:59.858944 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:59.858911 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:20:59.859961 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:20:59.859938 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:21:12.939483 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:12.939403 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerID="a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54" exitCode=0 Jul 30 11:21:12.939483 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:12.939450 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" event={"ID":"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19","Type":"ContainerDied","Data":"a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54"} Jul 30 11:21:12.939929 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:12.939866 2575 scope.go:117] "RemoveContainer" containerID="a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54" Jul 30 11:21:12.940879 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:12.940851 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:21:12.940944 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:12.940917 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:21:12.940981 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:12.940955 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:21:13.953835 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:13.953791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" event={"ID":"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19","Type":"ContainerStarted","Data":"06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73"} Jul 30 11:21:14.341110 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:14.339472 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:21:14.341110 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:14.340792 2575 patch_prober.go:28] interesting pod/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 30 11:21:14.341110 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:14.340847 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 30 11:21:14.858928 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:14.858896 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:21:15.339138 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:15.339073 2575 patch_prober.go:28] interesting pod/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 30 11:21:15.339630 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:15.339162 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 30 11:21:16.339339 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:16.339298 2575 patch_prober.go:28] interesting pod/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 30 11:21:16.339856 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:16.339363 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 30 11:21:17.338632 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:17.338591 2575 patch_prober.go:28] interesting pod/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 30 11:21:17.338961 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:17.338928 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 30 11:21:18.339130 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:18.339064 2575 patch_prober.go:28] interesting pod/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 30 11:21:18.339633 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:18.339153 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 30 11:21:19.342406 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:19.342378 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:21:19.343428 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:19.343405 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:21:25.107630 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:25.107596 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4"] Jul 30 11:21:25.118130 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:25.118081 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" Jul 30 11:21:25.120683 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:25.120661 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:21:25.132314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:25.132292 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4"] Jul 30 11:21:25.134506 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:25.134480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183b6fe-4b10-4e2e-9291-cc402a0fc24b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4\" (UID: \"4183b6fe-4b10-4e2e-9291-cc402a0fc24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" Jul 30 11:21:25.235433 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:25.235404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183b6fe-4b10-4e2e-9291-cc402a0fc24b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4\" (UID: \"4183b6fe-4b10-4e2e-9291-cc402a0fc24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" Jul 30 11:21:25.236022 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:25.235997 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183b6fe-4b10-4e2e-9291-cc402a0fc24b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4\" (UID: \"4183b6fe-4b10-4e2e-9291-cc402a0fc24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" Jul 30 11:21:25.428284 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:25.428208 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" Jul 30 11:21:25.567310 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:25.567282 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4"] Jul 30 11:21:25.567922 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:21:25.567901 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4183b6fe_4b10_4e2e_9291_cc402a0fc24b.slice/crio-7732eacd17fdb4ffbe54c3e0f90fe7dce6151669c53ee7ee84398b2792db8f0e WatchSource:0}: Error finding container 7732eacd17fdb4ffbe54c3e0f90fe7dce6151669c53ee7ee84398b2792db8f0e: Status 404 returned error can't find the container with id 7732eacd17fdb4ffbe54c3e0f90fe7dce6151669c53ee7ee84398b2792db8f0e Jul 30 11:21:26.025898 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:26.025817 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" event={"ID":"4183b6fe-4b10-4e2e-9291-cc402a0fc24b","Type":"ContainerStarted","Data":"637872ef5a7b51f022526db52600472e2ac98e1794908031fb1a5cbdb5666597"} Jul 30 11:21:26.025898 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:26.025867 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" event={"ID":"4183b6fe-4b10-4e2e-9291-cc402a0fc24b","Type":"ContainerStarted","Data":"7732eacd17fdb4ffbe54c3e0f90fe7dce6151669c53ee7ee84398b2792db8f0e"} Jul 30 11:21:26.043238 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:26.043190 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" podStartSLOduration=1.043174191 podStartE2EDuration="1.043174191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:21:26.040573861 +0000 UTC m=+1994.013434907" watchObservedRunningTime="2026-07-30 11:21:26.043174191 +0000 UTC m=+1994.016035226" Jul 30 11:21:27.033246 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:27.033209 2575 generic.go:356] "Generic (PLEG): container finished" podID="4183b6fe-4b10-4e2e-9291-cc402a0fc24b" containerID="637872ef5a7b51f022526db52600472e2ac98e1794908031fb1a5cbdb5666597" exitCode=0 Jul 30 11:21:27.033646 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:27.033291 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" event={"ID":"4183b6fe-4b10-4e2e-9291-cc402a0fc24b","Type":"ContainerDied","Data":"637872ef5a7b51f022526db52600472e2ac98e1794908031fb1a5cbdb5666597"} Jul 30 11:21:28.172281 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:28.172254 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" Jul 30 11:21:28.262640 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:28.262611 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183b6fe-4b10-4e2e-9291-cc402a0fc24b-maas-api-service-ca\") pod \"4183b6fe-4b10-4e2e-9291-cc402a0fc24b\" (UID: \"4183b6fe-4b10-4e2e-9291-cc402a0fc24b\") " Jul 30 11:21:28.262997 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:28.262978 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4183b6fe-4b10-4e2e-9291-cc402a0fc24b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4183b6fe-4b10-4e2e-9291-cc402a0fc24b" (UID: "4183b6fe-4b10-4e2e-9291-cc402a0fc24b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:21:28.364218 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:28.364143 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4183b6fe-4b10-4e2e-9291-cc402a0fc24b-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:21:29.046567 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:29.046533 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" event={"ID":"4183b6fe-4b10-4e2e-9291-cc402a0fc24b","Type":"ContainerDied","Data":"7732eacd17fdb4ffbe54c3e0f90fe7dce6151669c53ee7ee84398b2792db8f0e"} Jul 30 11:21:29.046746 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:29.046584 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7732eacd17fdb4ffbe54c3e0f90fe7dce6151669c53ee7ee84398b2792db8f0e" Jul 30 11:21:29.046746 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:29.046544 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4" Jul 30 11:21:32.211226 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:32.211176 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4"] Jul 30 11:21:32.218692 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:32.218642 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-772a6e1a-8kpr4"] Jul 30 11:21:32.336724 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:32.336688 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25"] Jul 30 11:21:32.337230 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:32.337174 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" containerID="cri-o://06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73" gracePeriod=30 Jul 30 11:21:32.528434 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:32.528343 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4183b6fe-4b10-4e2e-9291-cc402a0fc24b" path="/var/lib/kubelet/pods/4183b6fe-4b10-4e2e-9291-cc402a0fc24b/volumes" Jul 30 11:21:34.345735 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:34.345698 2575 patch_prober.go:28] interesting pod/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 11:21:34.346145 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:34.345764 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 11:21:37.594890 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.594865 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:21:37.653832 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.653749 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2jwc\" (UniqueName: \"kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-kube-api-access-k2jwc\") pod \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " Jul 30 11:21:37.653832 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.653790 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-podinfo\") pod \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " Jul 30 11:21:37.654046 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.653836 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-socket\") pod \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " Jul 30 11:21:37.654046 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.653862 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-certs\") pod \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " Jul 30 11:21:37.654046 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.653891 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-credential-socket\") pod \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " Jul 30 11:21:37.654046 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.653920 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istiod-ca-cert\") pod \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " Jul 30 11:21:37.654046 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.653978 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-data\") pod \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " Jul 30 11:21:37.654046 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654009 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-token\") pod \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " Jul 30 11:21:37.654372 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654057 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-envoy\") pod \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\" (UID: \"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19\") " Jul 30 11:21:37.654372 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654192 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" (UID: "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:21:37.654372 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654224 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" (UID: "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:21:37.654372 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654201 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" (UID: "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:21:37.654372 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654343 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" (UID: "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:21:37.654653 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654536 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" (UID: "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:21:37.654653 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654632 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-credential-socket\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:21:37.654768 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654652 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-data\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:21:37.654768 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654667 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-socket\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:21:37.654768 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.654680 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-workload-certs\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:21:37.656462 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.656351 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" (UID: "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:21:37.656462 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.656416 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" (UID: "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:21:37.656647 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.656475 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-kube-api-access-k2jwc" (OuterVolumeSpecName: "kube-api-access-k2jwc") pod "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" (UID: "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19"). InnerVolumeSpecName "kube-api-access-k2jwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:21:37.657201 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.657176 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" (UID: "5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 11:21:37.755959 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.755927 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2jwc\" (UniqueName: \"kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-kube-api-access-k2jwc\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:21:37.755959 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.755955 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-podinfo\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:21:37.755959 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.755965 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istiod-ca-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:21:37.756220 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.755974 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-token\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:21:37.756220 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:37.755982 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19-istio-envoy\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:21:38.106284 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.106254 2575 generic.go:356] "Generic (PLEG): container finished" podID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerID="06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73" exitCode=0 Jul 30 11:21:38.106457 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.106304 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" Jul 30 11:21:38.106457 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.106320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" event={"ID":"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19","Type":"ContainerDied","Data":"06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73"} Jul 30 11:21:38.106457 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.106368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25" event={"ID":"5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19","Type":"ContainerDied","Data":"146c8852dac6aced82aa11d04b83e01da9ca8232974a4d0ba6f7b3b5f5d10b05"} Jul 30 11:21:38.106457 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.106388 2575 scope.go:117] "RemoveContainer" containerID="06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73" Jul 30 11:21:38.116811 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.116791 2575 scope.go:117] "RemoveContainer" containerID="a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54" Jul 30 11:21:38.126010 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.125987 2575 scope.go:117] "RemoveContainer" containerID="06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73" Jul 30 11:21:38.126413 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:21:38.126388 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73\": container with ID starting with 06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73 not found: ID does not exist" containerID="06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73" Jul 30 11:21:38.126495 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.126420 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73"} err="failed to get container status \"06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73\": rpc error: code = NotFound desc = could not find container \"06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73\": container with ID starting with 06d7b228d34d12e18200ccaa4ce7ea2300297ad9f614ee5a7f5a3e613cd30b73 not found: ID does not exist" Jul 30 11:21:38.126495 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.126439 2575 scope.go:117] "RemoveContainer" containerID="a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54" Jul 30 11:21:38.126666 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:21:38.126649 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54\": container with ID starting with a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54 not found: ID does not exist" containerID="a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54" Jul 30 11:21:38.126707 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.126671 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54"} err="failed to get container status \"a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54\": rpc error: code = NotFound desc = could not find container \"a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54\": container with ID starting with a3c6e0e968b8e4cc9d3394ab8437ee14c130e31b8137c7e94e4a368d612d2c54 not found: ID does not exist" Jul 30 11:21:38.128339 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.128319 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25"] Jul 30 11:21:38.135304 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.135282 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-772a6e1a-openshift-default-659f7df8d4-c5n25"] Jul 30 11:21:38.534599 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:21:38.534562 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" path="/var/lib/kubelet/pods/5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19/volumes" Jul 30 11:23:12.655368 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:12.655342 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:23:12.663278 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:12.663256 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:23:38.353444 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.353413 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6"] Jul 30 11:23:38.354014 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.353838 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" Jul 30 11:23:38.354014 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.353850 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" Jul 30 11:23:38.354014 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.353859 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" Jul 30 11:23:38.354014 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.353865 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" Jul 30 11:23:38.354014 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.353876 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4183b6fe-4b10-4e2e-9291-cc402a0fc24b" containerName="revoke-keys" Jul 30 11:23:38.354014 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.353883 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4183b6fe-4b10-4e2e-9291-cc402a0fc24b" containerName="revoke-keys" Jul 30 11:23:38.354014 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.353966 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4183b6fe-4b10-4e2e-9291-cc402a0fc24b" containerName="revoke-keys" Jul 30 11:23:38.354014 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.353975 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" Jul 30 11:23:38.354014 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.353982 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f10a3dc-ff94-4d2b-be0e-e6947e4cbc19" containerName="istio-proxy" Jul 30 11:23:38.357565 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.357545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:23:38.359427 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.359410 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-203e4c55\"" Jul 30 11:23:38.359525 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.359462 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-203e4c55-dockercfg-t42bl\"" Jul 30 11:23:38.369276 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.369249 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6"] Jul 30 11:23:38.388564 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.388538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxjwl\" (UniqueName: \"kubernetes.io/projected/bca5abfc-d109-4866-9ba2-16dd83bde02d-kube-api-access-rxjwl\") pod \"payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6\" (UID: \"bca5abfc-d109-4866-9ba2-16dd83bde02d\") " pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:23:38.388682 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.388597 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca5abfc-d109-4866-9ba2-16dd83bde02d-plugins-config-volume\") pod \"payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6\" (UID: \"bca5abfc-d109-4866-9ba2-16dd83bde02d\") " pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:23:38.489279 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.489234 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca5abfc-d109-4866-9ba2-16dd83bde02d-plugins-config-volume\") pod \"payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6\" (UID: \"bca5abfc-d109-4866-9ba2-16dd83bde02d\") " pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:23:38.489448 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.489363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxjwl\" (UniqueName: \"kubernetes.io/projected/bca5abfc-d109-4866-9ba2-16dd83bde02d-kube-api-access-rxjwl\") pod \"payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6\" (UID: \"bca5abfc-d109-4866-9ba2-16dd83bde02d\") " pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:23:38.489935 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.489917 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca5abfc-d109-4866-9ba2-16dd83bde02d-plugins-config-volume\") pod \"payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6\" (UID: \"bca5abfc-d109-4866-9ba2-16dd83bde02d\") " pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:23:38.496850 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.496822 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxjwl\" (UniqueName: \"kubernetes.io/projected/bca5abfc-d109-4866-9ba2-16dd83bde02d-kube-api-access-rxjwl\") pod \"payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6\" (UID: \"bca5abfc-d109-4866-9ba2-16dd83bde02d\") " pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:23:38.669773 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.669691 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:23:38.811884 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.811858 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6"] Jul 30 11:23:38.813449 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:23:38.813408 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca5abfc_d109_4866_9ba2_16dd83bde02d.slice/crio-a414de873ebd3918a0cfd5e7a285f57a4992e25ed289ea5e531ce99d2711b289 WatchSource:0}: Error finding container a414de873ebd3918a0cfd5e7a285f57a4992e25ed289ea5e531ce99d2711b289: Status 404 returned error can't find the container with id a414de873ebd3918a0cfd5e7a285f57a4992e25ed289ea5e531ce99d2711b289 Jul 30 11:23:38.834207 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:38.834182 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" event={"ID":"bca5abfc-d109-4866-9ba2-16dd83bde02d","Type":"ContainerStarted","Data":"a414de873ebd3918a0cfd5e7a285f57a4992e25ed289ea5e531ce99d2711b289"} Jul 30 11:23:39.842295 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:39.842260 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" event={"ID":"bca5abfc-d109-4866-9ba2-16dd83bde02d","Type":"ContainerStarted","Data":"495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda"} Jul 30 11:23:39.842727 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:39.842405 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:23:39.858387 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:39.858343 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" podStartSLOduration=1.8583283929999999 podStartE2EDuration="1.858328393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:23:39.85679879 +0000 UTC m=+2127.829659841" watchObservedRunningTime="2026-07-30 11:23:39.858328393 +0000 UTC m=+2127.831189428" Jul 30 11:23:50.851549 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:23:50.851519 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:24:02.985648 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:02.985604 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6"] Jul 30 11:24:02.986261 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:02.985926 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" podUID="bca5abfc-d109-4866-9ba2-16dd83bde02d" containerName="payload-processing" containerID="cri-o://495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda" gracePeriod=30 Jul 30 11:24:03.265984 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.265959 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:24:03.319560 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.319525 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxjwl\" (UniqueName: \"kubernetes.io/projected/bca5abfc-d109-4866-9ba2-16dd83bde02d-kube-api-access-rxjwl\") pod \"bca5abfc-d109-4866-9ba2-16dd83bde02d\" (UID: \"bca5abfc-d109-4866-9ba2-16dd83bde02d\") " Jul 30 11:24:03.319738 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.319595 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca5abfc-d109-4866-9ba2-16dd83bde02d-plugins-config-volume\") pod \"bca5abfc-d109-4866-9ba2-16dd83bde02d\" (UID: \"bca5abfc-d109-4866-9ba2-16dd83bde02d\") " Jul 30 11:24:03.320050 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.320021 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca5abfc-d109-4866-9ba2-16dd83bde02d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bca5abfc-d109-4866-9ba2-16dd83bde02d" (UID: "bca5abfc-d109-4866-9ba2-16dd83bde02d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:24:03.321710 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.321686 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca5abfc-d109-4866-9ba2-16dd83bde02d-kube-api-access-rxjwl" (OuterVolumeSpecName: "kube-api-access-rxjwl") pod "bca5abfc-d109-4866-9ba2-16dd83bde02d" (UID: "bca5abfc-d109-4866-9ba2-16dd83bde02d"). InnerVolumeSpecName "kube-api-access-rxjwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:24:03.420936 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.420904 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca5abfc-d109-4866-9ba2-16dd83bde02d-plugins-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:24:03.420936 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.420930 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxjwl\" (UniqueName: \"kubernetes.io/projected/bca5abfc-d109-4866-9ba2-16dd83bde02d-kube-api-access-rxjwl\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:24:03.995970 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.995937 2575 generic.go:356] "Generic (PLEG): container finished" podID="bca5abfc-d109-4866-9ba2-16dd83bde02d" containerID="495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda" exitCode=0 Jul 30 11:24:03.996462 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.995985 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" Jul 30 11:24:03.996462 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.996025 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" event={"ID":"bca5abfc-d109-4866-9ba2-16dd83bde02d","Type":"ContainerDied","Data":"495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda"} Jul 30 11:24:03.996462 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.996107 2575 scope.go:117] "RemoveContainer" containerID="495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda" Jul 30 11:24:03.996462 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:03.996109 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6" event={"ID":"bca5abfc-d109-4866-9ba2-16dd83bde02d","Type":"ContainerDied","Data":"a414de873ebd3918a0cfd5e7a285f57a4992e25ed289ea5e531ce99d2711b289"} Jul 30 11:24:04.006458 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:04.006442 2575 scope.go:117] "RemoveContainer" containerID="495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda" Jul 30 11:24:04.006709 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:24:04.006688 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda\": container with ID starting with 495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda not found: ID does not exist" containerID="495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda" Jul 30 11:24:04.006776 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:04.006717 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda"} err="failed to get container status \"495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda\": rpc error: code = NotFound desc = could not find container \"495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda\": container with ID starting with 495ee25d4bfb247fd8e87675069c6f527fdd368ce2b6b8dcf1ebf4edfec9bdda not found: ID does not exist" Jul 30 11:24:04.017731 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:04.017709 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6"] Jul 30 11:24:04.021255 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:04.021235 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-203e4c55-6cc457975d-6rhx6"] Jul 30 11:24:04.527854 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:04.527822 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bca5abfc-d109-4866-9ba2-16dd83bde02d" path="/var/lib/kubelet/pods/bca5abfc-d109-4866-9ba2-16dd83bde02d/volumes" Jul 30 11:24:51.158257 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.158222 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt"] Jul 30 11:24:51.158776 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.158433 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" podUID="272bf8cb-ec29-459e-92f3-e2af3f34415c" containerName="manager" containerID="cri-o://1a7030596ce875d4fefc98bb049cc2a1e957a24477aecb3e1e3230f7db476184" gracePeriod=10 Jul 30 11:24:51.282226 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.282195 2575 generic.go:356] "Generic (PLEG): container finished" podID="272bf8cb-ec29-459e-92f3-e2af3f34415c" containerID="1a7030596ce875d4fefc98bb049cc2a1e957a24477aecb3e1e3230f7db476184" exitCode=0 Jul 30 11:24:51.282369 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.282268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" event={"ID":"272bf8cb-ec29-459e-92f3-e2af3f34415c","Type":"ContainerDied","Data":"1a7030596ce875d4fefc98bb049cc2a1e957a24477aecb3e1e3230f7db476184"} Jul 30 11:24:51.422242 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.422180 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:24:51.568057 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.568026 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggmbs\" (UniqueName: \"kubernetes.io/projected/272bf8cb-ec29-459e-92f3-e2af3f34415c-kube-api-access-ggmbs\") pod \"272bf8cb-ec29-459e-92f3-e2af3f34415c\" (UID: \"272bf8cb-ec29-459e-92f3-e2af3f34415c\") " Jul 30 11:24:51.568258 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.568132 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/272bf8cb-ec29-459e-92f3-e2af3f34415c-extensions-socket-volume\") pod \"272bf8cb-ec29-459e-92f3-e2af3f34415c\" (UID: \"272bf8cb-ec29-459e-92f3-e2af3f34415c\") " Jul 30 11:24:51.568466 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.568444 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272bf8cb-ec29-459e-92f3-e2af3f34415c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "272bf8cb-ec29-459e-92f3-e2af3f34415c" (UID: "272bf8cb-ec29-459e-92f3-e2af3f34415c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:24:51.570076 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.570046 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272bf8cb-ec29-459e-92f3-e2af3f34415c-kube-api-access-ggmbs" (OuterVolumeSpecName: "kube-api-access-ggmbs") pod "272bf8cb-ec29-459e-92f3-e2af3f34415c" (UID: "272bf8cb-ec29-459e-92f3-e2af3f34415c"). InnerVolumeSpecName "kube-api-access-ggmbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:24:51.669571 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.669537 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggmbs\" (UniqueName: \"kubernetes.io/projected/272bf8cb-ec29-459e-92f3-e2af3f34415c-kube-api-access-ggmbs\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:24:51.669571 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:51.669563 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/272bf8cb-ec29-459e-92f3-e2af3f34415c-extensions-socket-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:24:52.289873 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:52.289843 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" Jul 30 11:24:52.290342 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:52.289876 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt" event={"ID":"272bf8cb-ec29-459e-92f3-e2af3f34415c","Type":"ContainerDied","Data":"7583f7bd5963cba23e2e9735090f29a8cb9618f79c81caee6ddf75357eb9bd77"} Jul 30 11:24:52.290342 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:52.289938 2575 scope.go:117] "RemoveContainer" containerID="1a7030596ce875d4fefc98bb049cc2a1e957a24477aecb3e1e3230f7db476184" Jul 30 11:24:52.311871 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:52.311847 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt"] Jul 30 11:24:52.315528 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:52.315504 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-kxfvt"] Jul 30 11:24:52.529216 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:24:52.529189 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272bf8cb-ec29-459e-92f3-e2af3f34415c" path="/var/lib/kubelet/pods/272bf8cb-ec29-459e-92f3-e2af3f34415c/volumes" Jul 30 11:26:06.762308 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.762273 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz"] Jul 30 11:26:06.762733 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.762690 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272bf8cb-ec29-459e-92f3-e2af3f34415c" containerName="manager" Jul 30 11:26:06.762733 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.762701 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="272bf8cb-ec29-459e-92f3-e2af3f34415c" containerName="manager" Jul 30 11:26:06.762733 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.762712 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bca5abfc-d109-4866-9ba2-16dd83bde02d" containerName="payload-processing" Jul 30 11:26:06.762733 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.762718 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bca5abfc-d109-4866-9ba2-16dd83bde02d" containerName="payload-processing" Jul 30 11:26:06.762868 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.762784 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="272bf8cb-ec29-459e-92f3-e2af3f34415c" containerName="manager" Jul 30 11:26:06.762868 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.762795 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bca5abfc-d109-4866-9ba2-16dd83bde02d" containerName="payload-processing" Jul 30 11:26:06.765895 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.765878 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" Jul 30 11:26:06.768130 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.768110 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fqxbm\"" Jul 30 11:26:06.777775 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.777748 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz"] Jul 30 11:26:06.855498 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.855464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-mk2vz\" (UID: \"ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" Jul 30 11:26:06.855656 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.855512 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445q6\" (UniqueName: \"kubernetes.io/projected/ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755-kube-api-access-445q6\") pod \"kuadrant-operator-controller-manager-68cf589589-mk2vz\" (UID: \"ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" Jul 30 11:26:06.956113 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.956063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-mk2vz\" (UID: \"ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" Jul 30 11:26:06.956273 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.956210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445q6\" (UniqueName: \"kubernetes.io/projected/ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755-kube-api-access-445q6\") pod \"kuadrant-operator-controller-manager-68cf589589-mk2vz\" (UID: \"ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" Jul 30 11:26:06.956531 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.956508 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-mk2vz\" (UID: \"ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" Jul 30 11:26:06.964285 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:06.964267 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-445q6\" (UniqueName: \"kubernetes.io/projected/ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755-kube-api-access-445q6\") pod \"kuadrant-operator-controller-manager-68cf589589-mk2vz\" (UID: \"ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" Jul 30 11:26:07.076665 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:07.076633 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" Jul 30 11:26:07.215700 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:07.215674 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz"] Jul 30 11:26:07.217678 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:26:07.217650 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5fe991_c2f0_4ce5_aa89_4fae2bc2c755.slice/crio-4076aa86fb78c5228ae9225c96adc45a9c74e26be90388c08b3fb5d0ca24ec64 WatchSource:0}: Error finding container 4076aa86fb78c5228ae9225c96adc45a9c74e26be90388c08b3fb5d0ca24ec64: Status 404 returned error can't find the container with id 4076aa86fb78c5228ae9225c96adc45a9c74e26be90388c08b3fb5d0ca24ec64 Jul 30 11:26:07.723488 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:07.723457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" event={"ID":"ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755","Type":"ContainerStarted","Data":"dc96825038fced953176b1f8026255cdbc230b53b17fb71f062e4caf25ad5201"} Jul 30 11:26:07.723665 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:07.723515 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" Jul 30 11:26:07.723665 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:07.723532 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" event={"ID":"ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755","Type":"ContainerStarted","Data":"4076aa86fb78c5228ae9225c96adc45a9c74e26be90388c08b3fb5d0ca24ec64"} Jul 30 11:26:07.745759 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:07.745708 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" podStartSLOduration=1.74569245 podStartE2EDuration="1.74569245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:26:07.742312616 +0000 UTC m=+2275.715173680" watchObservedRunningTime="2026-07-30 11:26:07.74569245 +0000 UTC m=+2275.718553486" Jul 30 11:26:18.738543 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:18.738508 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-mk2vz" Jul 30 11:26:30.608224 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.608183 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs"] Jul 30 11:26:30.618657 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.618629 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:26:30.621274 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.621144 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6bccc947-dockercfg-gq6rn\"" Jul 30 11:26:30.621407 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.621345 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6bccc947\"" Jul 30 11:26:30.622512 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.622484 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs"] Jul 30 11:26:30.778937 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.778895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-plugins-config-volume\") pod \"payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs\" (UID: \"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac\") " pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:26:30.779161 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.778985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttt5\" (UniqueName: \"kubernetes.io/projected/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-kube-api-access-bttt5\") pod \"payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs\" (UID: \"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac\") " pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:26:30.880015 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.879921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-plugins-config-volume\") pod \"payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs\" (UID: \"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac\") " pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:26:30.880015 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.879996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttt5\" (UniqueName: \"kubernetes.io/projected/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-kube-api-access-bttt5\") pod \"payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs\" (UID: \"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac\") " pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:26:30.880704 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.880685 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-plugins-config-volume\") pod \"payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs\" (UID: \"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac\") " pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:26:30.891691 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.891669 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttt5\" (UniqueName: \"kubernetes.io/projected/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-kube-api-access-bttt5\") pod \"payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs\" (UID: \"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac\") " pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:26:30.932257 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:30.932231 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:26:31.075824 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:31.075793 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs"] Jul 30 11:26:31.077135 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:26:31.077057 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c697e8b_66a5_4e7d_93ce_70ebe1da5cac.slice/crio-21ca867d2546e7c02692c8d514e85e9d4b8a02c28c685d9580cfb014e543506d WatchSource:0}: Error finding container 21ca867d2546e7c02692c8d514e85e9d4b8a02c28c685d9580cfb014e543506d: Status 404 returned error can't find the container with id 21ca867d2546e7c02692c8d514e85e9d4b8a02c28c685d9580cfb014e543506d Jul 30 11:26:31.875374 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:31.875335 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" event={"ID":"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac","Type":"ContainerStarted","Data":"8f6bb3a6b2eafb59fcddb3636dcea2764ef8444823b22ba95d271433b6dc1e04"} Jul 30 11:26:31.875798 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:31.875398 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" event={"ID":"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac","Type":"ContainerStarted","Data":"21ca867d2546e7c02692c8d514e85e9d4b8a02c28c685d9580cfb014e543506d"} Jul 30 11:26:31.875798 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:31.875521 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:26:31.899927 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:31.899871 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" podStartSLOduration=1.8998556359999998 podStartE2EDuration="1.899855636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:26:31.898397035 +0000 UTC m=+2299.871258096" watchObservedRunningTime="2026-07-30 11:26:31.899855636 +0000 UTC m=+2299.872716692" Jul 30 11:26:42.884132 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:26:42.884101 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:27:33.319704 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:33.319614 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g"] Jul 30 11:27:33.323500 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:33.323473 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" Jul 30 11:27:33.325370 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:33.325349 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:27:33.330746 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:33.330654 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g"] Jul 30 11:27:33.360654 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:33.360622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1761aab-ccfb-4578-a192-1961f99e0f74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6bccc947-2j99g\" (UID: \"c1761aab-ccfb-4578-a192-1961f99e0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" Jul 30 11:27:33.461264 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:33.461228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1761aab-ccfb-4578-a192-1961f99e0f74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6bccc947-2j99g\" (UID: \"c1761aab-ccfb-4578-a192-1961f99e0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" Jul 30 11:27:33.461844 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:33.461824 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1761aab-ccfb-4578-a192-1961f99e0f74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6bccc947-2j99g\" (UID: \"c1761aab-ccfb-4578-a192-1961f99e0f74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" Jul 30 11:27:33.636809 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:33.636713 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" Jul 30 11:27:33.781939 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:33.781915 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g"] Jul 30 11:27:33.784468 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:27:33.784438 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1761aab_ccfb_4578_a192_1961f99e0f74.slice/crio-233277497c176fc28314e4fbb8cd12d99ef87bd136f83ae4cfe520652fab4fa3 WatchSource:0}: Error finding container 233277497c176fc28314e4fbb8cd12d99ef87bd136f83ae4cfe520652fab4fa3: Status 404 returned error can't find the container with id 233277497c176fc28314e4fbb8cd12d99ef87bd136f83ae4cfe520652fab4fa3 Jul 30 11:27:34.237349 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:34.237252 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" event={"ID":"c1761aab-ccfb-4578-a192-1961f99e0f74","Type":"ContainerStarted","Data":"05840126d90a6ac7188b4a5a72828f092eef150c6284149159fcefef5dea59a4"} Jul 30 11:27:34.237349 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:34.237316 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" event={"ID":"c1761aab-ccfb-4578-a192-1961f99e0f74","Type":"ContainerStarted","Data":"233277497c176fc28314e4fbb8cd12d99ef87bd136f83ae4cfe520652fab4fa3"} Jul 30 11:27:34.253835 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:34.253782 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" podStartSLOduration=1.253766524 podStartE2EDuration="1.253766524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:27:34.251624533 +0000 UTC m=+2362.224485570" watchObservedRunningTime="2026-07-30 11:27:34.253766524 +0000 UTC m=+2362.226627558" Jul 30 11:27:35.246857 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:35.246819 2575 generic.go:356] "Generic (PLEG): container finished" podID="c1761aab-ccfb-4578-a192-1961f99e0f74" containerID="05840126d90a6ac7188b4a5a72828f092eef150c6284149159fcefef5dea59a4" exitCode=0 Jul 30 11:27:35.247314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:35.246899 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" event={"ID":"c1761aab-ccfb-4578-a192-1961f99e0f74","Type":"ContainerDied","Data":"05840126d90a6ac7188b4a5a72828f092eef150c6284149159fcefef5dea59a4"} Jul 30 11:27:36.394276 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:36.394250 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" Jul 30 11:27:36.482887 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:36.482860 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1761aab-ccfb-4578-a192-1961f99e0f74-maas-api-service-ca\") pod \"c1761aab-ccfb-4578-a192-1961f99e0f74\" (UID: \"c1761aab-ccfb-4578-a192-1961f99e0f74\") " Jul 30 11:27:36.483248 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:36.483228 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1761aab-ccfb-4578-a192-1961f99e0f74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1761aab-ccfb-4578-a192-1961f99e0f74" (UID: "c1761aab-ccfb-4578-a192-1961f99e0f74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:27:36.583757 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:36.583724 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1761aab-ccfb-4578-a192-1961f99e0f74-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:27:37.260512 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:37.260484 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" Jul 30 11:27:37.260686 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:37.260519 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g" event={"ID":"c1761aab-ccfb-4578-a192-1961f99e0f74","Type":"ContainerDied","Data":"233277497c176fc28314e4fbb8cd12d99ef87bd136f83ae4cfe520652fab4fa3"} Jul 30 11:27:37.260686 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:37.260573 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="233277497c176fc28314e4fbb8cd12d99ef87bd136f83ae4cfe520652fab4fa3" Jul 30 11:27:43.143895 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.143859 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs"] Jul 30 11:27:43.144385 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.144108 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" podUID="5c697e8b-66a5-4e7d-93ce-70ebe1da5cac" containerName="payload-processing" containerID="cri-o://8f6bb3a6b2eafb59fcddb3636dcea2764ef8444823b22ba95d271433b6dc1e04" gracePeriod=30 Jul 30 11:27:43.298791 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.298761 2575 generic.go:356] "Generic (PLEG): container finished" podID="5c697e8b-66a5-4e7d-93ce-70ebe1da5cac" containerID="8f6bb3a6b2eafb59fcddb3636dcea2764ef8444823b22ba95d271433b6dc1e04" exitCode=0 Jul 30 11:27:43.298959 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.298825 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" event={"ID":"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac","Type":"ContainerDied","Data":"8f6bb3a6b2eafb59fcddb3636dcea2764ef8444823b22ba95d271433b6dc1e04"} Jul 30 11:27:43.305466 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.305433 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g"] Jul 30 11:27:43.308697 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.308676 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6bccc947-2j99g"] Jul 30 11:27:43.403846 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.403791 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:27:43.444256 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.444231 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bttt5\" (UniqueName: \"kubernetes.io/projected/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-kube-api-access-bttt5\") pod \"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac\" (UID: \"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac\") " Jul 30 11:27:43.444435 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.444271 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-plugins-config-volume\") pod \"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac\" (UID: \"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac\") " Jul 30 11:27:43.444718 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.444692 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c697e8b-66a5-4e7d-93ce-70ebe1da5cac" (UID: "5c697e8b-66a5-4e7d-93ce-70ebe1da5cac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:27:43.446341 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.446314 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-kube-api-access-bttt5" (OuterVolumeSpecName: "kube-api-access-bttt5") pod "5c697e8b-66a5-4e7d-93ce-70ebe1da5cac" (UID: "5c697e8b-66a5-4e7d-93ce-70ebe1da5cac"). InnerVolumeSpecName "kube-api-access-bttt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:27:43.545930 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.545893 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bttt5\" (UniqueName: \"kubernetes.io/projected/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-kube-api-access-bttt5\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:27:43.545930 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:43.545923 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac-plugins-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:27:44.306837 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:44.306797 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" Jul 30 11:27:44.307434 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:44.306845 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs" event={"ID":"5c697e8b-66a5-4e7d-93ce-70ebe1da5cac","Type":"ContainerDied","Data":"21ca867d2546e7c02692c8d514e85e9d4b8a02c28c685d9580cfb014e543506d"} Jul 30 11:27:44.307434 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:44.306924 2575 scope.go:117] "RemoveContainer" containerID="8f6bb3a6b2eafb59fcddb3636dcea2764ef8444823b22ba95d271433b6dc1e04" Jul 30 11:27:44.330164 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:44.330136 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs"] Jul 30 11:27:44.335433 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:44.335410 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6bccc947-769f758d78-hdbqs"] Jul 30 11:27:44.527841 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:44.527811 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c697e8b-66a5-4e7d-93ce-70ebe1da5cac" path="/var/lib/kubelet/pods/5c697e8b-66a5-4e7d-93ce-70ebe1da5cac/volumes" Jul 30 11:27:44.528154 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:27:44.528142 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1761aab-ccfb-4578-a192-1961f99e0f74" path="/var/lib/kubelet/pods/c1761aab-ccfb-4578-a192-1961f99e0f74/volumes" Jul 30 11:28:12.695765 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:28:12.695735 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:28:12.704868 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:28:12.704847 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:28:12.833896 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:28:12.833860 2575 scope.go:117] "RemoveContainer" containerID="637872ef5a7b51f022526db52600472e2ac98e1794908031fb1a5cbdb5666597" Jul 30 11:29:52.836190 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.836153 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs"] Jul 30 11:29:52.836852 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.836832 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1761aab-ccfb-4578-a192-1961f99e0f74" containerName="revoke-keys" Jul 30 11:29:52.836951 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.836855 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1761aab-ccfb-4578-a192-1961f99e0f74" containerName="revoke-keys" Jul 30 11:29:52.836951 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.836898 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c697e8b-66a5-4e7d-93ce-70ebe1da5cac" containerName="payload-processing" Jul 30 11:29:52.836951 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.836908 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c697e8b-66a5-4e7d-93ce-70ebe1da5cac" containerName="payload-processing" Jul 30 11:29:52.837134 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.837016 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c697e8b-66a5-4e7d-93ce-70ebe1da5cac" containerName="payload-processing" Jul 30 11:29:52.837134 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.837029 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1761aab-ccfb-4578-a192-1961f99e0f74" containerName="revoke-keys" Jul 30 11:29:52.840397 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.840375 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:29:52.842825 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.842805 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-da4b33\"" Jul 30 11:29:52.842956 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.842938 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-da4b33-dockercfg-jgx6f\"" Jul 30 11:29:52.850898 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.850874 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs"] Jul 30 11:29:52.908937 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.908906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140dba8b-ed99-43be-8d4f-415b89102192-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs\" (UID: \"140dba8b-ed99-43be-8d4f-415b89102192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:29:52.909164 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:52.908957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9wv2\" (UniqueName: \"kubernetes.io/projected/140dba8b-ed99-43be-8d4f-415b89102192-kube-api-access-r9wv2\") pod \"payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs\" (UID: \"140dba8b-ed99-43be-8d4f-415b89102192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:29:53.009770 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:53.009738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140dba8b-ed99-43be-8d4f-415b89102192-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs\" (UID: \"140dba8b-ed99-43be-8d4f-415b89102192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:29:53.009967 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:53.009793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9wv2\" (UniqueName: \"kubernetes.io/projected/140dba8b-ed99-43be-8d4f-415b89102192-kube-api-access-r9wv2\") pod \"payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs\" (UID: \"140dba8b-ed99-43be-8d4f-415b89102192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:29:53.010559 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:53.010531 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140dba8b-ed99-43be-8d4f-415b89102192-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs\" (UID: \"140dba8b-ed99-43be-8d4f-415b89102192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:29:53.018591 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:53.018570 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9wv2\" (UniqueName: \"kubernetes.io/projected/140dba8b-ed99-43be-8d4f-415b89102192-kube-api-access-r9wv2\") pod \"payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs\" (UID: \"140dba8b-ed99-43be-8d4f-415b89102192\") " pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:29:53.151901 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:53.151816 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:29:53.297207 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:53.297176 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs"] Jul 30 11:29:53.298680 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:29:53.298649 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140dba8b_ed99_43be_8d4f_415b89102192.slice/crio-3b361e0a2b4d77ffc1eb068e2435af3bf4ea627fd281ca08ccd88c8bec886c9c WatchSource:0}: Error finding container 3b361e0a2b4d77ffc1eb068e2435af3bf4ea627fd281ca08ccd88c8bec886c9c: Status 404 returned error can't find the container with id 3b361e0a2b4d77ffc1eb068e2435af3bf4ea627fd281ca08ccd88c8bec886c9c Jul 30 11:29:54.083996 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:54.083959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" event={"ID":"140dba8b-ed99-43be-8d4f-415b89102192","Type":"ContainerStarted","Data":"8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332"} Jul 30 11:29:54.084413 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:54.084026 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" event={"ID":"140dba8b-ed99-43be-8d4f-415b89102192","Type":"ContainerStarted","Data":"3b361e0a2b4d77ffc1eb068e2435af3bf4ea627fd281ca08ccd88c8bec886c9c"} Jul 30 11:29:54.084413 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:54.084149 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:29:54.103686 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:29:54.103642 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" podStartSLOduration=2.103628816 podStartE2EDuration="2.103628816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:29:54.100911462 +0000 UTC m=+2502.073772498" watchObservedRunningTime="2026-07-30 11:29:54.103628816 +0000 UTC m=+2502.076489853" Jul 30 11:30:00.133235 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:00.133193 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc"] Jul 30 11:30:00.137237 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:00.137210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc" Jul 30 11:30:00.139591 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:00.139571 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:30:00.147794 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:00.147761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc" Jul 30 11:30:00.149980 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:00.149949 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc"] Jul 30 11:30:00.285148 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:00.285123 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc"] Jul 30 11:30:00.287320 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:30:00.287294 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b03b48_2f79_4c20_a520_84907d4f3374.slice/crio-4f59e73ee6925acdb7d945ebebf90ed742022f57b4a478e60b6119c7ac5a54a9 WatchSource:0}: Error finding container 4f59e73ee6925acdb7d945ebebf90ed742022f57b4a478e60b6119c7ac5a54a9: Status 404 returned error can't find the container with id 4f59e73ee6925acdb7d945ebebf90ed742022f57b4a478e60b6119c7ac5a54a9 Jul 30 11:30:01.126829 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:01.126782 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc" event={"ID":"e3b03b48-2f79-4c20-a520-84907d4f3374","Type":"ContainerStarted","Data":"2d2c54dfd0d7d508c240d7260dcf02c692c52906613d3ba1d53d49681cca6e1c"} Jul 30 11:30:01.127016 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:01.126887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc" event={"ID":"e3b03b48-2f79-4c20-a520-84907d4f3374","Type":"ContainerStarted","Data":"4f59e73ee6925acdb7d945ebebf90ed742022f57b4a478e60b6119c7ac5a54a9"} Jul 30 11:30:01.147246 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:01.147194 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc" podStartSLOduration=1.147180204 podStartE2EDuration="1.147180204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:30:01.145390757 +0000 UTC m=+2509.118251826" watchObservedRunningTime="2026-07-30 11:30:01.147180204 +0000 UTC m=+2509.120041240" Jul 30 11:30:02.134193 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:02.134154 2575 generic.go:356] "Generic (PLEG): container finished" podID="e3b03b48-2f79-4c20-a520-84907d4f3374" containerID="2d2c54dfd0d7d508c240d7260dcf02c692c52906613d3ba1d53d49681cca6e1c" exitCode=0 Jul 30 11:30:02.134377 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:02.134243 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc" event={"ID":"e3b03b48-2f79-4c20-a520-84907d4f3374","Type":"ContainerDied","Data":"2d2c54dfd0d7d508c240d7260dcf02c692c52906613d3ba1d53d49681cca6e1c"} Jul 30 11:30:03.285722 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:03.285695 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc" Jul 30 11:30:04.147737 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:04.147706 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc" Jul 30 11:30:04.147937 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:04.147757 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc" event={"ID":"e3b03b48-2f79-4c20-a520-84907d4f3374","Type":"ContainerDied","Data":"4f59e73ee6925acdb7d945ebebf90ed742022f57b4a478e60b6119c7ac5a54a9"} Jul 30 11:30:04.147937 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:04.147809 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f59e73ee6925acdb7d945ebebf90ed742022f57b4a478e60b6119c7ac5a54a9" Jul 30 11:30:05.092226 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:05.092195 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:30:21.269881 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.269841 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6"] Jul 30 11:30:21.270405 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.270375 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3b03b48-2f79-4c20-a520-84907d4f3374" containerName="cleanup" Jul 30 11:30:21.270405 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.270390 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3b03b48-2f79-4c20-a520-84907d4f3374" containerName="cleanup" Jul 30 11:30:21.270525 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.270464 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3b03b48-2f79-4c20-a520-84907d4f3374" containerName="cleanup" Jul 30 11:30:21.273564 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.273548 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.275932 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.275915 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cac774-serving-cert\"" Jul 30 11:30:21.286082 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.286059 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6"] Jul 30 11:30:21.376418 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.376386 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28r8b\" (UniqueName: \"kubernetes.io/projected/13067273-9df4-41dc-9518-b83c27b09704-kube-api-access-28r8b\") pod \"maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.376576 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.376437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13067273-9df4-41dc-9518-b83c27b09704-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.376576 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.376486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13067273-9df4-41dc-9518-b83c27b09704-maas-api-tls\") pod \"maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.477466 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.477431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13067273-9df4-41dc-9518-b83c27b09704-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.477466 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.477468 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13067273-9df4-41dc-9518-b83c27b09704-maas-api-tls\") pod \"maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.477668 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.477546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28r8b\" (UniqueName: \"kubernetes.io/projected/13067273-9df4-41dc-9518-b83c27b09704-kube-api-access-28r8b\") pod \"maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.478112 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.478059 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13067273-9df4-41dc-9518-b83c27b09704-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.480017 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.479987 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13067273-9df4-41dc-9518-b83c27b09704-maas-api-tls\") pod \"maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.485845 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.485822 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28r8b\" (UniqueName: \"kubernetes.io/projected/13067273-9df4-41dc-9518-b83c27b09704-kube-api-access-28r8b\") pod \"maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.584337 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.584302 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:21.725423 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.725393 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6"] Jul 30 11:30:21.727302 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:30:21.727274 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13067273_9df4_41dc_9518_b83c27b09704.slice/crio-5b939770da070bbe97fe47fbcf19fa639baf55631469af6dfc16f09a94d8d22a WatchSource:0}: Error finding container 5b939770da070bbe97fe47fbcf19fa639baf55631469af6dfc16f09a94d8d22a: Status 404 returned error can't find the container with id 5b939770da070bbe97fe47fbcf19fa639baf55631469af6dfc16f09a94d8d22a Jul 30 11:30:21.729028 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.729001 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 11:30:21.967812 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.967715 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s"] Jul 30 11:30:21.972649 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.972624 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:30:21.974714 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.974692 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cac774\"" Jul 30 11:30:21.974843 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.974802 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cac774-dockercfg-mk8kx\"" Jul 30 11:30:21.982392 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:21.982372 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s"] Jul 30 11:30:22.084957 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.084920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsk5v\" (UniqueName: \"kubernetes.io/projected/a45eeeca-9b61-4481-9fb9-40e6c31158ed-kube-api-access-dsk5v\") pod \"payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s\" (UID: \"a45eeeca-9b61-4481-9fb9-40e6c31158ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:30:22.085181 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.084964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a45eeeca-9b61-4481-9fb9-40e6c31158ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s\" (UID: \"a45eeeca-9b61-4481-9fb9-40e6c31158ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:30:22.186434 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.186407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsk5v\" (UniqueName: \"kubernetes.io/projected/a45eeeca-9b61-4481-9fb9-40e6c31158ed-kube-api-access-dsk5v\") pod \"payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s\" (UID: \"a45eeeca-9b61-4481-9fb9-40e6c31158ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:30:22.186615 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.186445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a45eeeca-9b61-4481-9fb9-40e6c31158ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s\" (UID: \"a45eeeca-9b61-4481-9fb9-40e6c31158ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:30:22.187142 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.187124 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a45eeeca-9b61-4481-9fb9-40e6c31158ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s\" (UID: \"a45eeeca-9b61-4481-9fb9-40e6c31158ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:30:22.195107 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.195062 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsk5v\" (UniqueName: \"kubernetes.io/projected/a45eeeca-9b61-4481-9fb9-40e6c31158ed-kube-api-access-dsk5v\") pod \"payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s\" (UID: \"a45eeeca-9b61-4481-9fb9-40e6c31158ed\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:30:22.254179 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.254141 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" event={"ID":"13067273-9df4-41dc-9518-b83c27b09704","Type":"ContainerStarted","Data":"5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df"} Jul 30 11:30:22.254358 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.254209 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:22.254358 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.254231 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" event={"ID":"13067273-9df4-41dc-9518-b83c27b09704","Type":"ContainerStarted","Data":"5b939770da070bbe97fe47fbcf19fa639baf55631469af6dfc16f09a94d8d22a"} Jul 30 11:30:22.272790 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.272329 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" podStartSLOduration=0.838121684 podStartE2EDuration="1.27230772s" totalImagesPullingTime="434.186036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:30:22.271913869 +0000 UTC m=+2530.244774928" watchObservedRunningTime="2026-07-30 11:30:22.27230772 +0000 UTC m=+2530.245168756" Jul 30 11:30:22.285069 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.285037 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:30:22.433215 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:22.433179 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s"] Jul 30 11:30:22.434755 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:30:22.434731 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45eeeca_9b61_4481_9fb9_40e6c31158ed.slice/crio-3f4d4a90411f45107d356d17dd861ca732b38e6d798eaca2321338bf58a08fca WatchSource:0}: Error finding container 3f4d4a90411f45107d356d17dd861ca732b38e6d798eaca2321338bf58a08fca: Status 404 returned error can't find the container with id 3f4d4a90411f45107d356d17dd861ca732b38e6d798eaca2321338bf58a08fca Jul 30 11:30:23.261876 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:23.261837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" event={"ID":"a45eeeca-9b61-4481-9fb9-40e6c31158ed","Type":"ContainerStarted","Data":"8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3"} Jul 30 11:30:23.262067 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:23.261907 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" event={"ID":"a45eeeca-9b61-4481-9fb9-40e6c31158ed","Type":"ContainerStarted","Data":"3f4d4a90411f45107d356d17dd861ca732b38e6d798eaca2321338bf58a08fca"} Jul 30 11:30:23.283948 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:23.283895 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" podStartSLOduration=2.283874795 podStartE2EDuration="2.283874795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:30:23.281341612 +0000 UTC m=+2531.254202648" watchObservedRunningTime="2026-07-30 11:30:23.283874795 +0000 UTC m=+2531.256735831" Jul 30 11:30:24.267503 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:24.267468 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:30:28.267575 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:28.267548 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:30:35.275462 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:30:35.275429 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:31:55.925953 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:55.925859 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql"] Jul 30 11:31:55.929745 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:55.929724 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" Jul 30 11:31:55.931753 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:55.931730 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:31:55.938200 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:55.937960 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql"] Jul 30 11:31:55.975961 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:55.975929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/883d8572-8936-4357-a97a-db73d3b18690-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-da4b33-glfql\" (UID: \"883d8572-8936-4357-a97a-db73d3b18690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" Jul 30 11:31:56.077016 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:56.076975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/883d8572-8936-4357-a97a-db73d3b18690-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-da4b33-glfql\" (UID: \"883d8572-8936-4357-a97a-db73d3b18690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" Jul 30 11:31:56.077615 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:56.077593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/883d8572-8936-4357-a97a-db73d3b18690-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-da4b33-glfql\" (UID: \"883d8572-8936-4357-a97a-db73d3b18690\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" Jul 30 11:31:56.243694 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:56.243612 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" Jul 30 11:31:56.387941 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:56.387909 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql"] Jul 30 11:31:56.390132 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:31:56.390068 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883d8572_8936_4357_a97a_db73d3b18690.slice/crio-489699d6a6b4d0b352eecdb4d2a464bdcc6015373b330c69ad8960344d5813d1 WatchSource:0}: Error finding container 489699d6a6b4d0b352eecdb4d2a464bdcc6015373b330c69ad8960344d5813d1: Status 404 returned error can't find the container with id 489699d6a6b4d0b352eecdb4d2a464bdcc6015373b330c69ad8960344d5813d1 Jul 30 11:31:56.838862 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:56.838828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" event={"ID":"883d8572-8936-4357-a97a-db73d3b18690","Type":"ContainerStarted","Data":"3dc57c0ece0b2112f33bfd5d57b4b96aa5a4301e359e6a8434ac607e12468a73"} Jul 30 11:31:56.839071 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:56.838883 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" event={"ID":"883d8572-8936-4357-a97a-db73d3b18690","Type":"ContainerStarted","Data":"489699d6a6b4d0b352eecdb4d2a464bdcc6015373b330c69ad8960344d5813d1"} Jul 30 11:31:56.855109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:56.855050 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" podStartSLOduration=1.855034608 podStartE2EDuration="1.855034608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:31:56.852951563 +0000 UTC m=+2624.825812600" watchObservedRunningTime="2026-07-30 11:31:56.855034608 +0000 UTC m=+2624.827895642" Jul 30 11:31:57.846584 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:57.846550 2575 generic.go:356] "Generic (PLEG): container finished" podID="883d8572-8936-4357-a97a-db73d3b18690" containerID="3dc57c0ece0b2112f33bfd5d57b4b96aa5a4301e359e6a8434ac607e12468a73" exitCode=0 Jul 30 11:31:57.846969 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:57.846608 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" event={"ID":"883d8572-8936-4357-a97a-db73d3b18690","Type":"ContainerDied","Data":"3dc57c0ece0b2112f33bfd5d57b4b96aa5a4301e359e6a8434ac607e12468a73"} Jul 30 11:31:58.992113 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:58.992072 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" Jul 30 11:31:59.106107 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:59.106018 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/883d8572-8936-4357-a97a-db73d3b18690-maas-api-service-ca\") pod \"883d8572-8936-4357-a97a-db73d3b18690\" (UID: \"883d8572-8936-4357-a97a-db73d3b18690\") " Jul 30 11:31:59.106354 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:59.106331 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883d8572-8936-4357-a97a-db73d3b18690-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "883d8572-8936-4357-a97a-db73d3b18690" (UID: "883d8572-8936-4357-a97a-db73d3b18690"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:31:59.207666 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:59.207640 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/883d8572-8936-4357-a97a-db73d3b18690-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:31:59.859481 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:59.859452 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" Jul 30 11:31:59.859656 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:59.859481 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql" event={"ID":"883d8572-8936-4357-a97a-db73d3b18690","Type":"ContainerDied","Data":"489699d6a6b4d0b352eecdb4d2a464bdcc6015373b330c69ad8960344d5813d1"} Jul 30 11:31:59.859656 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:31:59.859526 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489699d6a6b4d0b352eecdb4d2a464bdcc6015373b330c69ad8960344d5813d1" Jul 30 11:32:06.528769 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.528737 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc"] Jul 30 11:32:06.530213 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.530178 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-da4b33-29756850-4xzlc"] Jul 30 11:32:06.555212 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.555182 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs"] Jul 30 11:32:06.555438 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.555413 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" podUID="140dba8b-ed99-43be-8d4f-415b89102192" containerName="payload-processing" containerID="cri-o://8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332" gracePeriod=30 Jul 30 11:32:06.728565 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.727913 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql"] Jul 30 11:32:06.731037 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.731011 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-da4b33-glfql"] Jul 30 11:32:06.827296 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.827273 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:32:06.902805 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.902773 2575 generic.go:356] "Generic (PLEG): container finished" podID="140dba8b-ed99-43be-8d4f-415b89102192" containerID="8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332" exitCode=0 Jul 30 11:32:06.902944 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.902816 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" Jul 30 11:32:06.902944 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.902823 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" event={"ID":"140dba8b-ed99-43be-8d4f-415b89102192","Type":"ContainerDied","Data":"8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332"} Jul 30 11:32:06.902944 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.902891 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs" event={"ID":"140dba8b-ed99-43be-8d4f-415b89102192","Type":"ContainerDied","Data":"3b361e0a2b4d77ffc1eb068e2435af3bf4ea627fd281ca08ccd88c8bec886c9c"} Jul 30 11:32:06.902944 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.902913 2575 scope.go:117] "RemoveContainer" containerID="8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332" Jul 30 11:32:06.912318 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.912295 2575 scope.go:117] "RemoveContainer" containerID="8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332" Jul 30 11:32:06.912577 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:32:06.912559 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332\": container with ID starting with 8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332 not found: ID does not exist" containerID="8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332" Jul 30 11:32:06.912653 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.912586 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332"} err="failed to get container status \"8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332\": rpc error: code = NotFound desc = could not find container \"8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332\": container with ID starting with 8bf914ba4fee0b9eb5e96aa01067ffd1ba92137392043390e132efe8f0099332 not found: ID does not exist" Jul 30 11:32:06.990205 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.990180 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9wv2\" (UniqueName: \"kubernetes.io/projected/140dba8b-ed99-43be-8d4f-415b89102192-kube-api-access-r9wv2\") pod \"140dba8b-ed99-43be-8d4f-415b89102192\" (UID: \"140dba8b-ed99-43be-8d4f-415b89102192\") " Jul 30 11:32:06.990375 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.990228 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140dba8b-ed99-43be-8d4f-415b89102192-plugins-config-volume\") pod \"140dba8b-ed99-43be-8d4f-415b89102192\" (UID: \"140dba8b-ed99-43be-8d4f-415b89102192\") " Jul 30 11:32:06.990607 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.990581 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140dba8b-ed99-43be-8d4f-415b89102192-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "140dba8b-ed99-43be-8d4f-415b89102192" (UID: "140dba8b-ed99-43be-8d4f-415b89102192"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:32:06.992186 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:06.992160 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140dba8b-ed99-43be-8d4f-415b89102192-kube-api-access-r9wv2" (OuterVolumeSpecName: "kube-api-access-r9wv2") pod "140dba8b-ed99-43be-8d4f-415b89102192" (UID: "140dba8b-ed99-43be-8d4f-415b89102192"). InnerVolumeSpecName "kube-api-access-r9wv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:32:07.091948 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:07.091877 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140dba8b-ed99-43be-8d4f-415b89102192-plugins-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:32:07.091948 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:07.091905 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9wv2\" (UniqueName: \"kubernetes.io/projected/140dba8b-ed99-43be-8d4f-415b89102192-kube-api-access-r9wv2\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:32:07.235710 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:07.235675 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs"] Jul 30 11:32:07.244002 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:07.243969 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-da4b33-84c97d6768-r95gs"] Jul 30 11:32:08.528012 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:08.527976 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="140dba8b-ed99-43be-8d4f-415b89102192" path="/var/lib/kubelet/pods/140dba8b-ed99-43be-8d4f-415b89102192/volumes" Jul 30 11:32:08.528506 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:08.528487 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883d8572-8936-4357-a97a-db73d3b18690" path="/var/lib/kubelet/pods/883d8572-8936-4357-a97a-db73d3b18690/volumes" Jul 30 11:32:08.528847 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:08.528832 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3b03b48-2f79-4c20-a520-84907d4f3374" path="/var/lib/kubelet/pods/e3b03b48-2f79-4c20-a520-84907d4f3374/volumes" Jul 30 11:32:13.992533 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:13.992492 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd"] Jul 30 11:32:13.993144 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:13.993125 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883d8572-8936-4357-a97a-db73d3b18690" containerName="revoke-keys" Jul 30 11:32:13.993144 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:13.993145 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="883d8572-8936-4357-a97a-db73d3b18690" containerName="revoke-keys" Jul 30 11:32:13.993370 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:13.993167 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140dba8b-ed99-43be-8d4f-415b89102192" containerName="payload-processing" Jul 30 11:32:13.993370 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:13.993176 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="140dba8b-ed99-43be-8d4f-415b89102192" containerName="payload-processing" Jul 30 11:32:13.993370 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:13.993315 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="140dba8b-ed99-43be-8d4f-415b89102192" containerName="payload-processing" Jul 30 11:32:13.993370 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:13.993332 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="883d8572-8936-4357-a97a-db73d3b18690" containerName="revoke-keys" Jul 30 11:32:13.998295 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:13.998273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" Jul 30 11:32:14.000418 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:14.000393 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:32:14.003918 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:14.003897 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd"] Jul 30 11:32:14.163797 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:14.163763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b2ea08-ee4e-490a-af74-6db0f21c9bd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd\" (UID: \"42b2ea08-ee4e-490a-af74-6db0f21c9bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" Jul 30 11:32:14.265190 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:14.265069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b2ea08-ee4e-490a-af74-6db0f21c9bd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd\" (UID: \"42b2ea08-ee4e-490a-af74-6db0f21c9bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" Jul 30 11:32:14.265735 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:14.265710 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b2ea08-ee4e-490a-af74-6db0f21c9bd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd\" (UID: \"42b2ea08-ee4e-490a-af74-6db0f21c9bd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" Jul 30 11:32:14.309964 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:14.309939 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" Jul 30 11:32:14.449948 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:14.449917 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd"] Jul 30 11:32:14.451360 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:32:14.451326 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b2ea08_ee4e_490a_af74_6db0f21c9bd5.slice/crio-ffc7a3a6b85eaebc7e030a8e6f7e476a6298527db8950e0296848db4e9a66607 WatchSource:0}: Error finding container ffc7a3a6b85eaebc7e030a8e6f7e476a6298527db8950e0296848db4e9a66607: Status 404 returned error can't find the container with id ffc7a3a6b85eaebc7e030a8e6f7e476a6298527db8950e0296848db4e9a66607 Jul 30 11:32:14.952521 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:14.952429 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" event={"ID":"42b2ea08-ee4e-490a-af74-6db0f21c9bd5","Type":"ContainerStarted","Data":"8e9c3914b5deac16298fb8df6afb8fa503d27a24db51b1176f3dadf092296315"} Jul 30 11:32:14.952521 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:14.952485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" event={"ID":"42b2ea08-ee4e-490a-af74-6db0f21c9bd5","Type":"ContainerStarted","Data":"ffc7a3a6b85eaebc7e030a8e6f7e476a6298527db8950e0296848db4e9a66607"} Jul 30 11:32:14.969000 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:14.968951 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" podStartSLOduration=1.968934543 podStartE2EDuration="1.968934543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:32:14.966901898 +0000 UTC m=+2642.939762933" watchObservedRunningTime="2026-07-30 11:32:14.968934543 +0000 UTC m=+2642.941795579" Jul 30 11:32:15.959586 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:15.959553 2575 generic.go:356] "Generic (PLEG): container finished" podID="42b2ea08-ee4e-490a-af74-6db0f21c9bd5" containerID="8e9c3914b5deac16298fb8df6afb8fa503d27a24db51b1176f3dadf092296315" exitCode=0 Jul 30 11:32:15.959976 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:15.959645 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" event={"ID":"42b2ea08-ee4e-490a-af74-6db0f21c9bd5","Type":"ContainerDied","Data":"8e9c3914b5deac16298fb8df6afb8fa503d27a24db51b1176f3dadf092296315"} Jul 30 11:32:17.103492 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:17.103468 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" Jul 30 11:32:17.295239 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:17.295208 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b2ea08-ee4e-490a-af74-6db0f21c9bd5-maas-api-service-ca\") pod \"42b2ea08-ee4e-490a-af74-6db0f21c9bd5\" (UID: \"42b2ea08-ee4e-490a-af74-6db0f21c9bd5\") " Jul 30 11:32:17.295560 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:17.295539 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b2ea08-ee4e-490a-af74-6db0f21c9bd5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42b2ea08-ee4e-490a-af74-6db0f21c9bd5" (UID: "42b2ea08-ee4e-490a-af74-6db0f21c9bd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:32:17.396078 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:17.396045 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b2ea08-ee4e-490a-af74-6db0f21c9bd5-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:32:17.974149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:17.974118 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" Jul 30 11:32:17.974149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:17.974141 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd" event={"ID":"42b2ea08-ee4e-490a-af74-6db0f21c9bd5","Type":"ContainerDied","Data":"ffc7a3a6b85eaebc7e030a8e6f7e476a6298527db8950e0296848db4e9a66607"} Jul 30 11:32:17.974358 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:17.974186 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc7a3a6b85eaebc7e030a8e6f7e476a6298527db8950e0296848db4e9a66607" Jul 30 11:32:24.517051 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.517010 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6"] Jul 30 11:32:24.517711 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.517634 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" podUID="13067273-9df4-41dc-9518-b83c27b09704" containerName="maas-api" containerID="cri-o://5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df" gracePeriod=30 Jul 30 11:32:24.610381 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.610346 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s"] Jul 30 11:32:24.610587 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.610550 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" podUID="a45eeeca-9b61-4481-9fb9-40e6c31158ed" containerName="payload-pre-processing" containerID="cri-o://8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3" gracePeriod=30 Jul 30 11:32:24.879314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.879290 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:32:24.893439 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.893410 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd"] Jul 30 11:32:24.898820 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.898802 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:32:24.906010 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.905984 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cac774-gk8jd"] Jul 30 11:32:24.961996 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.961963 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsk5v\" (UniqueName: \"kubernetes.io/projected/a45eeeca-9b61-4481-9fb9-40e6c31158ed-kube-api-access-dsk5v\") pod \"a45eeeca-9b61-4481-9fb9-40e6c31158ed\" (UID: \"a45eeeca-9b61-4481-9fb9-40e6c31158ed\") " Jul 30 11:32:24.962195 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.962049 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13067273-9df4-41dc-9518-b83c27b09704-maas-api-tls\") pod \"13067273-9df4-41dc-9518-b83c27b09704\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " Jul 30 11:32:24.962195 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.962079 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a45eeeca-9b61-4481-9fb9-40e6c31158ed-plugins-config-volume\") pod \"a45eeeca-9b61-4481-9fb9-40e6c31158ed\" (UID: \"a45eeeca-9b61-4481-9fb9-40e6c31158ed\") " Jul 30 11:32:24.962195 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.962158 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13067273-9df4-41dc-9518-b83c27b09704-openshift-service-ca\") pod \"13067273-9df4-41dc-9518-b83c27b09704\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " Jul 30 11:32:24.962388 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.962211 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28r8b\" (UniqueName: \"kubernetes.io/projected/13067273-9df4-41dc-9518-b83c27b09704-kube-api-access-28r8b\") pod \"13067273-9df4-41dc-9518-b83c27b09704\" (UID: \"13067273-9df4-41dc-9518-b83c27b09704\") " Jul 30 11:32:24.962458 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.962432 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45eeeca-9b61-4481-9fb9-40e6c31158ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a45eeeca-9b61-4481-9fb9-40e6c31158ed" (UID: "a45eeeca-9b61-4481-9fb9-40e6c31158ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:32:24.962539 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.962515 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13067273-9df4-41dc-9518-b83c27b09704-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13067273-9df4-41dc-9518-b83c27b09704" (UID: "13067273-9df4-41dc-9518-b83c27b09704"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:32:24.962595 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.962540 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a45eeeca-9b61-4481-9fb9-40e6c31158ed-plugins-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:32:24.964080 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.964053 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45eeeca-9b61-4481-9fb9-40e6c31158ed-kube-api-access-dsk5v" (OuterVolumeSpecName: "kube-api-access-dsk5v") pod "a45eeeca-9b61-4481-9fb9-40e6c31158ed" (UID: "a45eeeca-9b61-4481-9fb9-40e6c31158ed"). InnerVolumeSpecName "kube-api-access-dsk5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:32:24.964507 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.964481 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13067273-9df4-41dc-9518-b83c27b09704-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13067273-9df4-41dc-9518-b83c27b09704" (UID: "13067273-9df4-41dc-9518-b83c27b09704"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 11:32:24.964620 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:24.964570 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13067273-9df4-41dc-9518-b83c27b09704-kube-api-access-28r8b" (OuterVolumeSpecName: "kube-api-access-28r8b") pod "13067273-9df4-41dc-9518-b83c27b09704" (UID: "13067273-9df4-41dc-9518-b83c27b09704"). InnerVolumeSpecName "kube-api-access-28r8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:32:25.017744 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.017716 2575 generic.go:356] "Generic (PLEG): container finished" podID="a45eeeca-9b61-4481-9fb9-40e6c31158ed" containerID="8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3" exitCode=0 Jul 30 11:32:25.017875 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.017759 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" Jul 30 11:32:25.017875 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.017801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" event={"ID":"a45eeeca-9b61-4481-9fb9-40e6c31158ed","Type":"ContainerDied","Data":"8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3"} Jul 30 11:32:25.017875 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.017850 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s" event={"ID":"a45eeeca-9b61-4481-9fb9-40e6c31158ed","Type":"ContainerDied","Data":"3f4d4a90411f45107d356d17dd861ca732b38e6d798eaca2321338bf58a08fca"} Jul 30 11:32:25.017875 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.017875 2575 scope.go:117] "RemoveContainer" containerID="8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3" Jul 30 11:32:25.019229 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.019203 2575 generic.go:356] "Generic (PLEG): container finished" podID="13067273-9df4-41dc-9518-b83c27b09704" containerID="5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df" exitCode=0 Jul 30 11:32:25.019339 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.019256 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" Jul 30 11:32:25.019400 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.019257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" event={"ID":"13067273-9df4-41dc-9518-b83c27b09704","Type":"ContainerDied","Data":"5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df"} Jul 30 11:32:25.019448 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.019406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6" event={"ID":"13067273-9df4-41dc-9518-b83c27b09704","Type":"ContainerDied","Data":"5b939770da070bbe97fe47fbcf19fa639baf55631469af6dfc16f09a94d8d22a"} Jul 30 11:32:25.029000 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.028969 2575 scope.go:117] "RemoveContainer" containerID="8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3" Jul 30 11:32:25.029314 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:32:25.029288 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3\": container with ID starting with 8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3 not found: ID does not exist" containerID="8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3" Jul 30 11:32:25.029314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.029323 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3"} err="failed to get container status \"8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3\": rpc error: code = NotFound desc = could not find container \"8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3\": container with ID starting with 8c6aec30a10df31eff6b0ce4693e4243b0e54e15561b5faaa2e1c27e643c76e3 not found: ID does not exist" Jul 30 11:32:25.029516 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.029366 2575 scope.go:117] "RemoveContainer" containerID="5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df" Jul 30 11:32:25.039059 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.039041 2575 scope.go:117] "RemoveContainer" containerID="5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df" Jul 30 11:32:25.039318 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:32:25.039297 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df\": container with ID starting with 5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df not found: ID does not exist" containerID="5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df" Jul 30 11:32:25.039381 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.039324 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df"} err="failed to get container status \"5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df\": rpc error: code = NotFound desc = could not find container \"5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df\": container with ID starting with 5cdf2b352b89b94604e873bc89e495c6449dba25c2b0b28bc56233ce0ca8e4df not found: ID does not exist" Jul 30 11:32:25.047056 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.047033 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s"] Jul 30 11:32:25.053519 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.053500 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cac774-558ffc7bdd-hbk5s"] Jul 30 11:32:25.063485 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.063468 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13067273-9df4-41dc-9518-b83c27b09704-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:32:25.063561 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.063487 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13067273-9df4-41dc-9518-b83c27b09704-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:32:25.063561 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.063496 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28r8b\" (UniqueName: \"kubernetes.io/projected/13067273-9df4-41dc-9518-b83c27b09704-kube-api-access-28r8b\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:32:25.063561 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.063505 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsk5v\" (UniqueName: \"kubernetes.io/projected/a45eeeca-9b61-4481-9fb9-40e6c31158ed-kube-api-access-dsk5v\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:32:25.069899 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.069876 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6"] Jul 30 11:32:25.072077 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:25.072058 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cac774-7cf8d9f484-fhdb6"] Jul 30 11:32:26.527465 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:26.527426 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13067273-9df4-41dc-9518-b83c27b09704" path="/var/lib/kubelet/pods/13067273-9df4-41dc-9518-b83c27b09704/volumes" Jul 30 11:32:26.527981 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:26.527753 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b2ea08-ee4e-490a-af74-6db0f21c9bd5" path="/var/lib/kubelet/pods/42b2ea08-ee4e-490a-af74-6db0f21c9bd5/volumes" Jul 30 11:32:26.528068 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:26.528055 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a45eeeca-9b61-4481-9fb9-40e6c31158ed" path="/var/lib/kubelet/pods/a45eeeca-9b61-4481-9fb9-40e6c31158ed/volumes" Jul 30 11:32:32.538184 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.538152 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp"] Jul 30 11:32:32.538645 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.538603 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a45eeeca-9b61-4481-9fb9-40e6c31158ed" containerName="payload-pre-processing" Jul 30 11:32:32.538645 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.538615 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a45eeeca-9b61-4481-9fb9-40e6c31158ed" containerName="payload-pre-processing" Jul 30 11:32:32.538645 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.538626 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13067273-9df4-41dc-9518-b83c27b09704" containerName="maas-api" Jul 30 11:32:32.538645 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.538632 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="13067273-9df4-41dc-9518-b83c27b09704" containerName="maas-api" Jul 30 11:32:32.538645 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.538641 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b2ea08-ee4e-490a-af74-6db0f21c9bd5" containerName="revoke-keys" Jul 30 11:32:32.538810 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.538649 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b2ea08-ee4e-490a-af74-6db0f21c9bd5" containerName="revoke-keys" Jul 30 11:32:32.538810 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.538743 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a45eeeca-9b61-4481-9fb9-40e6c31158ed" containerName="payload-pre-processing" Jul 30 11:32:32.538810 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.538752 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="13067273-9df4-41dc-9518-b83c27b09704" containerName="maas-api" Jul 30 11:32:32.538810 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.538760 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b2ea08-ee4e-490a-af74-6db0f21c9bd5" containerName="revoke-keys" Jul 30 11:32:32.541998 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.541980 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.544168 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.544147 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d6bcbf-openshift-default-dockercfg-wws4j\"" Jul 30 11:32:32.553321 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.553284 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp"] Jul 30 11:32:32.636781 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.636743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9mql\" (UniqueName: \"kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-kube-api-access-p9mql\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.636951 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.636810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-podinfo\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.636951 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.636913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-socket\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.637043 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.636976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-certs\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.637043 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.637003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-token\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.637139 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.637077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-envoy\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.637180 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.637138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-data\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.637222 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.637183 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istiod-ca-cert\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.637277 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.637245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-credential-socket\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.738412 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.738368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-data\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.738412 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.738406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istiod-ca-cert\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.738674 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.738432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-credential-socket\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.738674 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.738597 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9mql\" (UniqueName: \"kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-kube-api-access-p9mql\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.738805 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.738696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-podinfo\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.738805 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.738754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-socket\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.738805 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.738784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-credential-socket\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.738805 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.738798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-certs\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.739025 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.738831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-token\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.739025 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.738868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-envoy\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.739149 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.739024 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-socket\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.739206 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.739186 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istiod-ca-cert\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.739316 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.739296 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-data\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.739424 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.739399 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-certs\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.741150 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.741126 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-envoy\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.741274 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.741259 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-podinfo\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.747649 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.747623 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-token\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.747810 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.747790 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9mql\" (UniqueName: \"kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-kube-api-access-p9mql\") pod \"e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.852966 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.852880 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:32.998954 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:32.998924 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp"] Jul 30 11:32:33.001866 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:32:33.001837 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b307e99_21ad_49c5_aa0a_2850cd6ab143.slice/crio-c1264a9759f9f4917baabc9c665b6cd64e912b86688612f54b4948f29dca879b WatchSource:0}: Error finding container c1264a9759f9f4917baabc9c665b6cd64e912b86688612f54b4948f29dca879b: Status 404 returned error can't find the container with id c1264a9759f9f4917baabc9c665b6cd64e912b86688612f54b4948f29dca879b Jul 30 11:32:33.003979 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:33.003943 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:32:33.004054 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:33.004034 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:32:33.004139 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:33.004120 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:32:33.072051 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:33.072018 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" event={"ID":"5b307e99-21ad-49c5-aa0a-2850cd6ab143","Type":"ContainerStarted","Data":"c1264a9759f9f4917baabc9c665b6cd64e912b86688612f54b4948f29dca879b"} Jul 30 11:32:34.079792 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:34.079755 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" event={"ID":"5b307e99-21ad-49c5-aa0a-2850cd6ab143","Type":"ContainerStarted","Data":"cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd"} Jul 30 11:32:34.099550 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:34.099494 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" podStartSLOduration=2.099476146 podStartE2EDuration="2.099476146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:32:34.098577478 +0000 UTC m=+2662.071438534" watchObservedRunningTime="2026-07-30 11:32:34.099476146 +0000 UTC m=+2662.072337184" Jul 30 11:32:34.853378 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:34.853344 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:34.859562 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:34.859529 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:35.086332 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:35.086300 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:35.087399 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:35.087380 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:50.551900 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:50.551857 2575 patch_prober.go:28] interesting pod/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": read tcp 10.132.0.2:58250->10.132.0.81:15021: read: connection reset by peer" start-of-body= Jul 30 11:32:50.552348 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:50.551936 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": read tcp 10.132.0.2:58250->10.132.0.81:15021: read: connection reset by peer" Jul 30 11:32:51.202289 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:51.202257 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerID="cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd" exitCode=0 Jul 30 11:32:51.202476 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:51.202328 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" event={"ID":"5b307e99-21ad-49c5-aa0a-2850cd6ab143","Type":"ContainerDied","Data":"cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd"} Jul 30 11:32:51.202727 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:51.202713 2575 scope.go:117] "RemoveContainer" containerID="cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd" Jul 30 11:32:51.203793 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:51.203769 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:32:51.203855 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:51.203839 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:32:51.203895 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:51.203872 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:32:52.210426 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:52.210392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" event={"ID":"5b307e99-21ad-49c5-aa0a-2850cd6ab143","Type":"ContainerStarted","Data":"dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4"} Jul 30 11:32:52.853357 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:52.853316 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:52.853621 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:52.853593 2575 patch_prober.go:28] interesting pod/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Jul 30 11:32:52.853691 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:52.853667 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Jul 30 11:32:53.853767 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:53.853731 2575 patch_prober.go:28] interesting pod/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Jul 30 11:32:53.854288 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:53.853796 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Jul 30 11:32:54.853687 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:54.853643 2575 patch_prober.go:28] interesting pod/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Jul 30 11:32:54.853912 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:54.853718 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Jul 30 11:32:55.854128 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:55.854064 2575 patch_prober.go:28] interesting pod/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Jul 30 11:32:55.854566 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:55.854149 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Jul 30 11:32:56.854061 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:56.854020 2575 patch_prober.go:28] interesting pod/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Jul 30 11:32:56.854303 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:56.854082 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Jul 30 11:32:57.857567 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:57.857538 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:57.857930 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:57.857862 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:32:57.858436 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:32:57.858420 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:33:06.771240 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:06.771148 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx"] Jul 30 11:33:06.774359 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:06.774337 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:33:06.776342 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:06.776314 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3c382f-dockercfg-fp25b\"" Jul 30 11:33:06.776450 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:06.776314 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3c382f\"" Jul 30 11:33:06.786440 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:06.786413 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx"] Jul 30 11:33:06.901053 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:06.901007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0057bc75-314e-4fdf-84e9-8fda01519cc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx\" (UID: \"0057bc75-314e-4fdf-84e9-8fda01519cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:33:06.901229 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:06.901186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978j5\" (UniqueName: \"kubernetes.io/projected/0057bc75-314e-4fdf-84e9-8fda01519cc3-kube-api-access-978j5\") pod \"payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx\" (UID: \"0057bc75-314e-4fdf-84e9-8fda01519cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:33:07.001968 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:07.001938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978j5\" (UniqueName: \"kubernetes.io/projected/0057bc75-314e-4fdf-84e9-8fda01519cc3-kube-api-access-978j5\") pod \"payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx\" (UID: \"0057bc75-314e-4fdf-84e9-8fda01519cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:33:07.002181 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:07.002051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0057bc75-314e-4fdf-84e9-8fda01519cc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx\" (UID: \"0057bc75-314e-4fdf-84e9-8fda01519cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:33:07.002725 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:07.002705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0057bc75-314e-4fdf-84e9-8fda01519cc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx\" (UID: \"0057bc75-314e-4fdf-84e9-8fda01519cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:33:07.010056 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:07.010031 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-978j5\" (UniqueName: \"kubernetes.io/projected/0057bc75-314e-4fdf-84e9-8fda01519cc3-kube-api-access-978j5\") pod \"payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx\" (UID: \"0057bc75-314e-4fdf-84e9-8fda01519cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:33:07.086575 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:07.086549 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:33:07.434394 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:07.434361 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx"] Jul 30 11:33:07.435003 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:33:07.434975 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0057bc75_314e_4fdf_84e9_8fda01519cc3.slice/crio-5f3d4b750bf5b1b8a2c3ebad0ff57f344693ae1926e54ee59b0a371ead20f3d8 WatchSource:0}: Error finding container 5f3d4b750bf5b1b8a2c3ebad0ff57f344693ae1926e54ee59b0a371ead20f3d8: Status 404 returned error can't find the container with id 5f3d4b750bf5b1b8a2c3ebad0ff57f344693ae1926e54ee59b0a371ead20f3d8 Jul 30 11:33:08.332659 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:08.332625 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" event={"ID":"0057bc75-314e-4fdf-84e9-8fda01519cc3","Type":"ContainerStarted","Data":"4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe"} Jul 30 11:33:08.333117 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:08.332687 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:33:08.333117 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:08.332706 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" event={"ID":"0057bc75-314e-4fdf-84e9-8fda01519cc3","Type":"ContainerStarted","Data":"5f3d4b750bf5b1b8a2c3ebad0ff57f344693ae1926e54ee59b0a371ead20f3d8"} Jul 30 11:33:08.363478 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:08.363431 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" podStartSLOduration=2.363412833 podStartE2EDuration="2.363412833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:33:08.361952786 +0000 UTC m=+2696.334813853" watchObservedRunningTime="2026-07-30 11:33:08.363412833 +0000 UTC m=+2696.336273867" Jul 30 11:33:12.739161 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:12.739136 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:33:12.752427 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:12.752405 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:33:19.342447 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:19.342413 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:33:55.734517 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:55.734474 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp"] Jul 30 11:33:55.735115 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:55.734731 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" containerID="cri-o://dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4" gracePeriod=30 Jul 30 11:33:57.860184 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:57.860145 2575 patch_prober.go:28] interesting pod/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 11:33:57.860567 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:33:57.860225 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 11:34:00.991997 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:00.991969 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:34:01.102144 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102109 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-socket\") pod \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " Jul 30 11:34:01.102326 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102162 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-credential-socket\") pod \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " Jul 30 11:34:01.102326 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102186 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-podinfo\") pod \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " Jul 30 11:34:01.102326 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102230 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-data\") pod \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " Jul 30 11:34:01.102326 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102254 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-envoy\") pod \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " Jul 30 11:34:01.102326 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102290 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9mql\" (UniqueName: \"kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-kube-api-access-p9mql\") pod \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " Jul 30 11:34:01.102326 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102312 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-token\") pod \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " Jul 30 11:34:01.102625 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102358 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-certs\") pod \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " Jul 30 11:34:01.102625 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102406 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istiod-ca-cert\") pod \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\" (UID: \"5b307e99-21ad-49c5-aa0a-2850cd6ab143\") " Jul 30 11:34:01.102625 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102418 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5b307e99-21ad-49c5-aa0a-2850cd6ab143" (UID: "5b307e99-21ad-49c5-aa0a-2850cd6ab143"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:34:01.102625 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102475 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5b307e99-21ad-49c5-aa0a-2850cd6ab143" (UID: "5b307e99-21ad-49c5-aa0a-2850cd6ab143"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:34:01.102625 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102543 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-data" (OuterVolumeSpecName: "istio-data") pod "5b307e99-21ad-49c5-aa0a-2850cd6ab143" (UID: "5b307e99-21ad-49c5-aa0a-2850cd6ab143"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:34:01.102876 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102741 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5b307e99-21ad-49c5-aa0a-2850cd6ab143" (UID: "5b307e99-21ad-49c5-aa0a-2850cd6ab143"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:34:01.102876 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102811 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-credential-socket\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:01.102876 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102831 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-data\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:01.102876 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102845 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-socket\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:01.102876 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.102841 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5b307e99-21ad-49c5-aa0a-2850cd6ab143" (UID: "5b307e99-21ad-49c5-aa0a-2850cd6ab143"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:34:01.104564 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.104529 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-token" (OuterVolumeSpecName: "istio-token") pod "5b307e99-21ad-49c5-aa0a-2850cd6ab143" (UID: "5b307e99-21ad-49c5-aa0a-2850cd6ab143"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:34:01.104674 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.104562 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-kube-api-access-p9mql" (OuterVolumeSpecName: "kube-api-access-p9mql") pod "5b307e99-21ad-49c5-aa0a-2850cd6ab143" (UID: "5b307e99-21ad-49c5-aa0a-2850cd6ab143"). InnerVolumeSpecName "kube-api-access-p9mql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:34:01.104674 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.104597 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5b307e99-21ad-49c5-aa0a-2850cd6ab143" (UID: "5b307e99-21ad-49c5-aa0a-2850cd6ab143"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 11:34:01.104674 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.104599 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5b307e99-21ad-49c5-aa0a-2850cd6ab143" (UID: "5b307e99-21ad-49c5-aa0a-2850cd6ab143"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:34:01.204385 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.204359 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-podinfo\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:01.204385 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.204383 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-envoy\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:01.204385 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.204392 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9mql\" (UniqueName: \"kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-kube-api-access-p9mql\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:01.204593 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.204400 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istio-token\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:01.204593 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.204411 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b307e99-21ad-49c5-aa0a-2850cd6ab143-workload-certs\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:01.204593 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.204419 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b307e99-21ad-49c5-aa0a-2850cd6ab143-istiod-ca-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:01.658881 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.658844 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerID="dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4" exitCode=0 Jul 30 11:34:01.659054 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.658892 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" Jul 30 11:34:01.659054 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.658922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" event={"ID":"5b307e99-21ad-49c5-aa0a-2850cd6ab143","Type":"ContainerDied","Data":"dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4"} Jul 30 11:34:01.659054 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.658970 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp" event={"ID":"5b307e99-21ad-49c5-aa0a-2850cd6ab143","Type":"ContainerDied","Data":"c1264a9759f9f4917baabc9c665b6cd64e912b86688612f54b4948f29dca879b"} Jul 30 11:34:01.659054 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.658994 2575 scope.go:117] "RemoveContainer" containerID="dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4" Jul 30 11:34:01.669069 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.669052 2575 scope.go:117] "RemoveContainer" containerID="cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd" Jul 30 11:34:01.677509 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.677481 2575 scope.go:117] "RemoveContainer" containerID="dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4" Jul 30 11:34:01.677854 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:34:01.677810 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4\": container with ID starting with dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4 not found: ID does not exist" containerID="dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4" Jul 30 11:34:01.677967 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.677862 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4"} err="failed to get container status \"dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4\": rpc error: code = NotFound desc = could not find container \"dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4\": container with ID starting with dab4d24b509523be2beb90b91561d9f4e33de57cbca4a4e270f51949933728d4 not found: ID does not exist" Jul 30 11:34:01.677967 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.677886 2575 scope.go:117] "RemoveContainer" containerID="cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd" Jul 30 11:34:01.678228 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:34:01.678207 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd\": container with ID starting with cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd not found: ID does not exist" containerID="cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd" Jul 30 11:34:01.678309 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.678238 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd"} err="failed to get container status \"cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd\": rpc error: code = NotFound desc = could not find container \"cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd\": container with ID starting with cf4ccff687297d68b1a2c5a72380de9df1e96441fb6970d386e0b8496241aebd not found: ID does not exist" Jul 30 11:34:01.680569 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.680551 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp"] Jul 30 11:34:01.686203 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:01.686181 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d6bcbf-openshift-default-557b6f5fbb-9xsbp"] Jul 30 11:34:02.527836 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:02.527800 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" path="/var/lib/kubelet/pods/5b307e99-21ad-49c5-aa0a-2850cd6ab143/volumes" Jul 30 11:34:11.755736 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:11.755696 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx"] Jul 30 11:34:11.756257 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:11.755956 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" podUID="0057bc75-314e-4fdf-84e9-8fda01519cc3" containerName="payload-pre-processing" containerID="cri-o://4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe" gracePeriod=30 Jul 30 11:34:12.016107 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.016039 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:34:12.116156 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.116127 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0057bc75-314e-4fdf-84e9-8fda01519cc3-plugins-config-volume\") pod \"0057bc75-314e-4fdf-84e9-8fda01519cc3\" (UID: \"0057bc75-314e-4fdf-84e9-8fda01519cc3\") " Jul 30 11:34:12.116362 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.116198 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-978j5\" (UniqueName: \"kubernetes.io/projected/0057bc75-314e-4fdf-84e9-8fda01519cc3-kube-api-access-978j5\") pod \"0057bc75-314e-4fdf-84e9-8fda01519cc3\" (UID: \"0057bc75-314e-4fdf-84e9-8fda01519cc3\") " Jul 30 11:34:12.116560 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.116534 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0057bc75-314e-4fdf-84e9-8fda01519cc3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0057bc75-314e-4fdf-84e9-8fda01519cc3" (UID: "0057bc75-314e-4fdf-84e9-8fda01519cc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:34:12.118274 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.118247 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0057bc75-314e-4fdf-84e9-8fda01519cc3-kube-api-access-978j5" (OuterVolumeSpecName: "kube-api-access-978j5") pod "0057bc75-314e-4fdf-84e9-8fda01519cc3" (UID: "0057bc75-314e-4fdf-84e9-8fda01519cc3"). InnerVolumeSpecName "kube-api-access-978j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:34:12.217656 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.217616 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0057bc75-314e-4fdf-84e9-8fda01519cc3-plugins-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:12.217656 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.217651 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-978j5\" (UniqueName: \"kubernetes.io/projected/0057bc75-314e-4fdf-84e9-8fda01519cc3-kube-api-access-978j5\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:34:12.732484 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.732450 2575 generic.go:356] "Generic (PLEG): container finished" podID="0057bc75-314e-4fdf-84e9-8fda01519cc3" containerID="4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe" exitCode=0 Jul 30 11:34:12.732655 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.732501 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" Jul 30 11:34:12.732655 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.732513 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" event={"ID":"0057bc75-314e-4fdf-84e9-8fda01519cc3","Type":"ContainerDied","Data":"4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe"} Jul 30 11:34:12.732655 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.732566 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx" event={"ID":"0057bc75-314e-4fdf-84e9-8fda01519cc3","Type":"ContainerDied","Data":"5f3d4b750bf5b1b8a2c3ebad0ff57f344693ae1926e54ee59b0a371ead20f3d8"} Jul 30 11:34:12.732655 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.732598 2575 scope.go:117] "RemoveContainer" containerID="4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe" Jul 30 11:34:12.742041 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.742018 2575 scope.go:117] "RemoveContainer" containerID="4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe" Jul 30 11:34:12.742340 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:34:12.742318 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe\": container with ID starting with 4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe not found: ID does not exist" containerID="4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe" Jul 30 11:34:12.742419 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.742354 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe"} err="failed to get container status \"4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe\": rpc error: code = NotFound desc = could not find container \"4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe\": container with ID starting with 4533fc6cf21b6000a659d629f11081eb1278d7885d08288dc81e458c4481edbe not found: ID does not exist" Jul 30 11:34:12.753074 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.753044 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx"] Jul 30 11:34:12.763168 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.763142 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3c382f-5bc478b676-frqcx"] Jul 30 11:34:12.923571 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:12.923491 2575 scope.go:117] "RemoveContainer" containerID="05840126d90a6ac7188b4a5a72828f092eef150c6284149159fcefef5dea59a4" Jul 30 11:34:14.527199 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:14.527161 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0057bc75-314e-4fdf-84e9-8fda01519cc3" path="/var/lib/kubelet/pods/0057bc75-314e-4fdf-84e9-8fda01519cc3/volumes" Jul 30 11:34:19.433910 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.433872 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22"] Jul 30 11:34:19.434532 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.434511 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" Jul 30 11:34:19.434624 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.434534 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" Jul 30 11:34:19.434624 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.434553 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0057bc75-314e-4fdf-84e9-8fda01519cc3" containerName="payload-pre-processing" Jul 30 11:34:19.434624 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.434561 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0057bc75-314e-4fdf-84e9-8fda01519cc3" containerName="payload-pre-processing" Jul 30 11:34:19.434624 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.434578 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" Jul 30 11:34:19.434624 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.434586 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" Jul 30 11:34:19.434868 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.434709 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0057bc75-314e-4fdf-84e9-8fda01519cc3" containerName="payload-pre-processing" Jul 30 11:34:19.434868 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.434722 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" Jul 30 11:34:19.434868 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.434735 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b307e99-21ad-49c5-aa0a-2850cd6ab143" containerName="istio-proxy" Jul 30 11:34:19.439716 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.439692 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.443279 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.443255 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-060e9f-openshift-default-dockercfg-5pm89\"" Jul 30 11:34:19.457621 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.457593 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22"] Jul 30 11:34:19.481791 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.481762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922c1710-1d7b-475a-b72d-708e7414cd49-istiod-ca-cert\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.481971 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.481800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-istio-token\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.481971 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.481831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-credential-socket\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.481971 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.481880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-socket\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.481971 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.481920 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-certs\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.482130 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.481971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-data\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.482130 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.481997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-envoy\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.482130 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.482027 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922c1710-1d7b-475a-b72d-708e7414cd49-istio-podinfo\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.482130 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.482051 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqjs\" (UniqueName: \"kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-kube-api-access-5hqjs\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583198 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583159 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-istio-token\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583388 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-credential-socket\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583388 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-socket\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583388 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583283 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-certs\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583388 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-data\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583388 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-envoy\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583388 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922c1710-1d7b-475a-b72d-708e7414cd49-istio-podinfo\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583727 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqjs\" (UniqueName: \"kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-kube-api-access-5hqjs\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583727 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922c1710-1d7b-475a-b72d-708e7414cd49-istiod-ca-cert\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583727 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583617 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-credential-socket\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583887 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-data\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583887 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-certs\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.583993 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.583886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-socket\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.584301 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.584283 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922c1710-1d7b-475a-b72d-708e7414cd49-istiod-ca-cert\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.585776 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.585757 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-envoy\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.586051 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.586031 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922c1710-1d7b-475a-b72d-708e7414cd49-istio-podinfo\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.593499 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.593477 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-istio-token\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.594996 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.594979 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqjs\" (UniqueName: \"kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-kube-api-access-5hqjs\") pod \"e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.750315 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.750210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:19.902482 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.902453 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22"] Jul 30 11:34:19.903772 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:34:19.903748 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922c1710_1d7b_475a_b72d_708e7414cd49.slice/crio-c982e53eb556033959d5a558e8003c0e9bface38e5e50282bce71c64d8352e25 WatchSource:0}: Error finding container c982e53eb556033959d5a558e8003c0e9bface38e5e50282bce71c64d8352e25: Status 404 returned error can't find the container with id c982e53eb556033959d5a558e8003c0e9bface38e5e50282bce71c64d8352e25 Jul 30 11:34:19.905805 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.905774 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:34:19.905896 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.905848 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:34:19.905896 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:19.905890 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:34:20.783185 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:20.783146 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" event={"ID":"922c1710-1d7b-475a-b72d-708e7414cd49","Type":"ContainerStarted","Data":"ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc"} Jul 30 11:34:20.783607 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:20.783221 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" event={"ID":"922c1710-1d7b-475a-b72d-708e7414cd49","Type":"ContainerStarted","Data":"c982e53eb556033959d5a558e8003c0e9bface38e5e50282bce71c64d8352e25"} Jul 30 11:34:20.809245 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:20.809200 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" podStartSLOduration=1.809173669 podStartE2EDuration="1.809173669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:34:20.806641414 +0000 UTC m=+2768.779502449" watchObservedRunningTime="2026-07-30 11:34:20.809173669 +0000 UTC m=+2768.782034703" Jul 30 11:34:21.750826 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:21.750753 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:21.755462 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:21.755436 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:21.789664 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:21.789637 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:21.790519 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:21.790502 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:36.957539 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:36.957500 2575 patch_prober.go:28] interesting pod/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": read tcp 10.132.0.2:39268->10.132.0.83:15021: read: connection reset by peer" start-of-body= Jul 30 11:34:36.957977 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:36.957586 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": read tcp 10.132.0.2:39268->10.132.0.83:15021: read: connection reset by peer" Jul 30 11:34:37.899293 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:37.899261 2575 generic.go:356] "Generic (PLEG): container finished" podID="922c1710-1d7b-475a-b72d-708e7414cd49" containerID="ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc" exitCode=0 Jul 30 11:34:37.899460 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:37.899328 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" event={"ID":"922c1710-1d7b-475a-b72d-708e7414cd49","Type":"ContainerDied","Data":"ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc"} Jul 30 11:34:37.899737 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:37.899712 2575 scope.go:117] "RemoveContainer" containerID="ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc" Jul 30 11:34:37.900805 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:37.900778 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:34:37.900893 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:37.900860 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:34:37.900941 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:37.900908 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:34:38.906448 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:38.906413 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" event={"ID":"922c1710-1d7b-475a-b72d-708e7414cd49","Type":"ContainerStarted","Data":"cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23"} Jul 30 11:34:39.751204 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:39.751115 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:39.751513 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:39.751487 2575 patch_prober.go:28] interesting pod/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 30 11:34:39.751664 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:39.751642 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 30 11:34:40.751313 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:40.751274 2575 patch_prober.go:28] interesting pod/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 30 11:34:40.751905 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:40.751869 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 30 11:34:41.751375 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:41.751334 2575 patch_prober.go:28] interesting pod/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 30 11:34:41.751989 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:41.751957 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 30 11:34:42.751448 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:42.751399 2575 patch_prober.go:28] interesting pod/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 30 11:34:42.751977 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:42.751472 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 30 11:34:43.751305 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:43.751273 2575 patch_prober.go:28] interesting pod/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 30 11:34:43.751490 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:43.751329 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 30 11:34:44.754141 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:44.754107 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:44.754579 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:44.754562 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:44.755230 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:44.755203 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:34:53.792687 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:53.792651 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk"] Jul 30 11:34:53.795470 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:53.795451 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:34:53.800351 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:53.800328 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-945db9\"" Jul 30 11:34:53.800481 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:53.800375 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-945db9-dockercfg-zfvtx\"" Jul 30 11:34:53.811735 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:53.811709 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk"] Jul 30 11:34:53.908743 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:53.908704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83631c3a-ce95-4120-ae91-3004c098c354-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk\" (UID: \"83631c3a-ce95-4120-ae91-3004c098c354\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:34:53.908910 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:53.908791 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j66t9\" (UniqueName: \"kubernetes.io/projected/83631c3a-ce95-4120-ae91-3004c098c354-kube-api-access-j66t9\") pod \"payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk\" (UID: \"83631c3a-ce95-4120-ae91-3004c098c354\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:34:54.009510 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:54.009477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j66t9\" (UniqueName: \"kubernetes.io/projected/83631c3a-ce95-4120-ae91-3004c098c354-kube-api-access-j66t9\") pod \"payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk\" (UID: \"83631c3a-ce95-4120-ae91-3004c098c354\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:34:54.009679 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:54.009537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83631c3a-ce95-4120-ae91-3004c098c354-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk\" (UID: \"83631c3a-ce95-4120-ae91-3004c098c354\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:34:54.010210 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:54.010192 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83631c3a-ce95-4120-ae91-3004c098c354-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk\" (UID: \"83631c3a-ce95-4120-ae91-3004c098c354\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:34:54.017400 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:54.017376 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j66t9\" (UniqueName: \"kubernetes.io/projected/83631c3a-ce95-4120-ae91-3004c098c354-kube-api-access-j66t9\") pod \"payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk\" (UID: \"83631c3a-ce95-4120-ae91-3004c098c354\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:34:54.106538 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:54.106453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:34:54.249931 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:54.249902 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk"] Jul 30 11:34:54.251234 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:34:54.251205 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83631c3a_ce95_4120_ae91_3004c098c354.slice/crio-3f76178198a0ac55e4bb4d2d6723c8c6a03b9670916edfe008033e04b8752b09 WatchSource:0}: Error finding container 3f76178198a0ac55e4bb4d2d6723c8c6a03b9670916edfe008033e04b8752b09: Status 404 returned error can't find the container with id 3f76178198a0ac55e4bb4d2d6723c8c6a03b9670916edfe008033e04b8752b09 Jul 30 11:34:55.009485 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:55.009449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" event={"ID":"83631c3a-ce95-4120-ae91-3004c098c354","Type":"ContainerStarted","Data":"ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0"} Jul 30 11:34:55.009874 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:55.009510 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" event={"ID":"83631c3a-ce95-4120-ae91-3004c098c354","Type":"ContainerStarted","Data":"3f76178198a0ac55e4bb4d2d6723c8c6a03b9670916edfe008033e04b8752b09"} Jul 30 11:34:55.009874 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:55.009552 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:34:55.026663 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:34:55.026611 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" podStartSLOduration=2.026598295 podStartE2EDuration="2.026598295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:34:55.025612641 +0000 UTC m=+2802.998473699" watchObservedRunningTime="2026-07-30 11:34:55.026598295 +0000 UTC m=+2802.999459348" Jul 30 11:35:06.017829 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:35:06.017795 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:36:12.964934 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:36:12.964857 2575 scope.go:117] "RemoveContainer" containerID="2d2c54dfd0d7d508c240d7260dcf02c692c52906613d3ba1d53d49681cca6e1c" Jul 30 11:38:12.791470 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:38:12.790424 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:38:12.802658 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:38:12.802636 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:38:12.996910 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:38:12.996877 2575 scope.go:117] "RemoveContainer" containerID="3dc57c0ece0b2112f33bfd5d57b4b96aa5a4301e359e6a8434ac607e12468a73" Jul 30 11:39:13.019791 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:39:13.019758 2575 scope.go:117] "RemoveContainer" containerID="8e9c3914b5deac16298fb8df6afb8fa503d27a24db51b1176f3dadf092296315" Jul 30 11:43:12.838632 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:12.838597 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:43:12.846438 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:12.846415 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:43:44.727508 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.727310 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52"] Jul 30 11:43:44.733800 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.733767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.737161 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.737133 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f0ce38-openshift-default-dockercfg-mbld6\"" Jul 30 11:43:44.749573 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.749545 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52"] Jul 30 11:43:44.880273 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.880237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-data\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.880273 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.880272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-token\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.880532 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.880291 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c68677-723b-4d0c-90b6-6ff222e192f5-istiod-ca-cert\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.880532 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.880369 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-certs\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.880532 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.880443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-socket\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.880532 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.880510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-podinfo\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.880749 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.880545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-credential-socket\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.880749 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.880583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-envoy\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.880749 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.880659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwt5w\" (UniqueName: \"kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-kube-api-access-gwt5w\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.981492 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-data\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.981492 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-token\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.981492 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c68677-723b-4d0c-90b6-6ff222e192f5-istiod-ca-cert\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.981492 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-certs\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.981824 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-socket\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.981824 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981559 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-podinfo\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.981824 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-credential-socket\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.981824 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-envoy\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.981824 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwt5w\" (UniqueName: \"kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-kube-api-access-gwt5w\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.982074 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981857 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-data\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.982074 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.981907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-socket\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.982207 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.982119 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-credential-socket\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.982207 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.982184 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-certs\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.982288 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.982253 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c68677-723b-4d0c-90b6-6ff222e192f5-istiod-ca-cert\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.983894 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.983871 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-envoy\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.984206 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.984187 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-podinfo\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.990475 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.990454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwt5w\" (UniqueName: \"kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-kube-api-access-gwt5w\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:44.990569 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:44.990463 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-token\") pod \"e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:45.047347 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:45.047319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:45.192886 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:45.192854 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52"] Jul 30 11:43:45.194048 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:43:45.194022 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c68677_723b_4d0c_90b6_6ff222e192f5.slice/crio-911a070bb5f5b0ab70e43b37d6012f2f706eb9b2666549140fbd2862ecb6d792 WatchSource:0}: Error finding container 911a070bb5f5b0ab70e43b37d6012f2f706eb9b2666549140fbd2862ecb6d792: Status 404 returned error can't find the container with id 911a070bb5f5b0ab70e43b37d6012f2f706eb9b2666549140fbd2862ecb6d792 Jul 30 11:43:45.196333 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:45.196290 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:43:45.196425 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:45.196362 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:43:45.196425 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:45.196393 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:43:45.249835 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:45.249804 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" event={"ID":"21c68677-723b-4d0c-90b6-6ff222e192f5","Type":"ContainerStarted","Data":"911a070bb5f5b0ab70e43b37d6012f2f706eb9b2666549140fbd2862ecb6d792"} Jul 30 11:43:46.259077 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:46.259038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" event={"ID":"21c68677-723b-4d0c-90b6-6ff222e192f5","Type":"ContainerStarted","Data":"f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf"} Jul 30 11:43:46.277589 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:46.277542 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" podStartSLOduration=2.277525497 podStartE2EDuration="2.277525497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:43:46.276153952 +0000 UTC m=+3334.249015007" watchObservedRunningTime="2026-07-30 11:43:46.277525497 +0000 UTC m=+3334.250386531" Jul 30 11:43:47.048509 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:47.048472 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:47.053786 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:47.053759 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:47.264989 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:47.264956 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:47.265838 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:47.265819 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:43:50.451474 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.451442 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z"] Jul 30 11:43:50.455473 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.455443 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:50.457350 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.457323 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f0ce38-serving-cert\"" Jul 30 11:43:50.466475 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.466017 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z"] Jul 30 11:43:50.536071 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.536037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51914b67-c18f-476a-9f34-d598210e5ee0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:50.536258 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.536106 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51914b67-c18f-476a-9f34-d598210e5ee0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:50.536258 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.536215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7cg9\" (UniqueName: \"kubernetes.io/projected/51914b67-c18f-476a-9f34-d598210e5ee0-kube-api-access-d7cg9\") pod \"maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:50.637304 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.637268 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7cg9\" (UniqueName: \"kubernetes.io/projected/51914b67-c18f-476a-9f34-d598210e5ee0-kube-api-access-d7cg9\") pod \"maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:50.637491 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.637361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51914b67-c18f-476a-9f34-d598210e5ee0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:50.637491 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.637410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51914b67-c18f-476a-9f34-d598210e5ee0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:50.637582 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:43:50.637555 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-serving-cert: secret "maas-api-e2e-ipp-b-f0ce38-serving-cert" not found Jul 30 11:43:50.637631 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:43:50.637618 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51914b67-c18f-476a-9f34-d598210e5ee0-maas-api-tls podName:51914b67-c18f-476a-9f34-d598210e5ee0 nodeName:}" failed. No retries permitted until 2026-07-30 11:43:51.137602747 +0000 UTC m=+3339.110463759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51914b67-c18f-476a-9f34-d598210e5ee0-maas-api-tls") pod "maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" (UID: "51914b67-c18f-476a-9f34-d598210e5ee0") : secret "maas-api-e2e-ipp-b-f0ce38-serving-cert" not found Jul 30 11:43:50.637980 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.637960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51914b67-c18f-476a-9f34-d598210e5ee0-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:50.646921 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:50.646898 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7cg9\" (UniqueName: \"kubernetes.io/projected/51914b67-c18f-476a-9f34-d598210e5ee0-kube-api-access-d7cg9\") pod \"maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:51.142190 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:51.142158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51914b67-c18f-476a-9f34-d598210e5ee0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:51.144573 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:51.144547 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51914b67-c18f-476a-9f34-d598210e5ee0-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:51.371961 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:51.371925 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:51.717841 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:51.717805 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z"] Jul 30 11:43:51.718210 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:43:51.717988 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51914b67_c18f_476a_9f34_d598210e5ee0.slice/crio-8ac3fba759c4892ef8e3dc4d20403d5a0f41a4044b593ce3aa108834f9a29013 WatchSource:0}: Error finding container 8ac3fba759c4892ef8e3dc4d20403d5a0f41a4044b593ce3aa108834f9a29013: Status 404 returned error can't find the container with id 8ac3fba759c4892ef8e3dc4d20403d5a0f41a4044b593ce3aa108834f9a29013 Jul 30 11:43:51.719371 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:51.719354 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 11:43:52.296278 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:52.296239 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" event={"ID":"51914b67-c18f-476a-9f34-d598210e5ee0","Type":"ContainerStarted","Data":"1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48"} Jul 30 11:43:52.296484 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:52.296301 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:43:52.296484 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:52.296327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" event={"ID":"51914b67-c18f-476a-9f34-d598210e5ee0","Type":"ContainerStarted","Data":"8ac3fba759c4892ef8e3dc4d20403d5a0f41a4044b593ce3aa108834f9a29013"} Jul 30 11:43:52.316469 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:52.316421 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" podStartSLOduration=1.895744659 podStartE2EDuration="2.316406697s" totalImagesPullingTime="420.662038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:43:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:43:52.314346008 +0000 UTC m=+3340.287207238" watchObservedRunningTime="2026-07-30 11:43:52.316406697 +0000 UTC m=+3340.289267744" Jul 30 11:43:58.308299 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:43:58.308266 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:44:02.489698 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:02.489304 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": read tcp 10.132.0.2:48958->10.132.0.85:15021: read: connection reset by peer" start-of-body= Jul 30 11:44:02.489698 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:02.489383 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": read tcp 10.132.0.2:48958->10.132.0.85:15021: read: connection reset by peer" Jul 30 11:44:03.372685 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:03.372653 2575 generic.go:356] "Generic (PLEG): container finished" podID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerID="f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf" exitCode=0 Jul 30 11:44:03.372852 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:03.372723 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" event={"ID":"21c68677-723b-4d0c-90b6-6ff222e192f5","Type":"ContainerDied","Data":"f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf"} Jul 30 11:44:03.373144 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:03.373129 2575 scope.go:117] "RemoveContainer" containerID="f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf" Jul 30 11:44:03.374215 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:03.374187 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:44:03.374294 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:03.374253 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:44:03.374294 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:03.374287 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 30 11:44:04.381246 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:04.381216 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" event={"ID":"21c68677-723b-4d0c-90b6-6ff222e192f5","Type":"ContainerStarted","Data":"df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48"} Jul 30 11:44:05.048703 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:05.048668 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:44:05.049122 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:05.049014 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Jul 30 11:44:05.049310 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:05.049287 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Jul 30 11:44:06.048272 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:06.048229 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Jul 30 11:44:06.048777 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:06.048314 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Jul 30 11:44:07.048628 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:07.048587 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Jul 30 11:44:07.049281 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:07.049243 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Jul 30 11:44:08.048502 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:08.048465 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Jul 30 11:44:08.048803 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:08.048770 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Jul 30 11:44:09.048290 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:09.048255 2575 patch_prober.go:28] interesting pod/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Jul 30 11:44:09.048584 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:09.048555 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Jul 30 11:44:09.901974 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:09.901943 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-w2xrv_baba2139-d918-4f6f-9eab-3d14183d9659/payload-processing/0.log" Jul 30 11:44:10.051656 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:10.051626 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:44:10.052025 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:10.052004 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:44:10.052606 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:44:10.052584 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:45:00.148173 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.148139 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9"] Jul 30 11:45:00.152015 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.151992 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9" Jul 30 11:45:00.154037 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.154013 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:45:00.154746 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.154724 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756865-zm68m"] Jul 30 11:45:00.158944 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.158914 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7"] Jul 30 11:45:00.159141 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.159108 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756865-zm68m" Jul 30 11:45:00.163071 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.163051 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7" Jul 30 11:45:00.163347 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.163326 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7"] Jul 30 11:45:00.166036 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.166016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9" Jul 30 11:45:00.168386 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.167625 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7" Jul 30 11:45:00.168386 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.167740 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp"] Jul 30 11:45:00.179025 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.179001 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9"] Jul 30 11:45:00.179264 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.179238 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756865-zm68m"] Jul 30 11:45:00.179516 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.179500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp" Jul 30 11:45:00.179948 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.179924 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7"] Jul 30 11:45:00.185235 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.185215 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp"] Jul 30 11:45:00.189157 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.189125 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7"] Jul 30 11:45:00.210688 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.210662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756865-zm68m" Jul 30 11:45:00.218782 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.218762 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7" Jul 30 11:45:00.276063 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.276005 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7" Jul 30 11:45:00.284197 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.284176 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp" Jul 30 11:45:00.441425 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:45:00.441355 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c074905_a0b3_4abf_aa4c_29effd535a07.slice/crio-cc04822a655339d575f8277c02ce40379fd5aa98d1730674e575d8d08fa16eae WatchSource:0}: Error finding container cc04822a655339d575f8277c02ce40379fd5aa98d1730674e575d8d08fa16eae: Status 404 returned error can't find the container with id cc04822a655339d575f8277c02ce40379fd5aa98d1730674e575d8d08fa16eae Jul 30 11:45:00.454196 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.450687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756865-zm68m"] Jul 30 11:45:00.472354 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.472321 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7"] Jul 30 11:45:00.476593 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:45:00.476530 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383e78c7_f033_4e92_9262_94d1d31ed65d.slice/crio-9f0c6eb57f357531636da7f7c4fa871de851950d2f46f84696590b25bf0ef010 WatchSource:0}: Error finding container 9f0c6eb57f357531636da7f7c4fa871de851950d2f46f84696590b25bf0ef010: Status 404 returned error can't find the container with id 9f0c6eb57f357531636da7f7c4fa871de851950d2f46f84696590b25bf0ef010 Jul 30 11:45:00.541299 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:45:00.541256 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad425334_0bc9_4ce2_a0fd_02dff8169f61.slice/crio-fb8b9107606df409fbab749630ca143a0c36ca361f47fd233761267191d1a6ab WatchSource:0}: Error finding container fb8b9107606df409fbab749630ca143a0c36ca361f47fd233761267191d1a6ab: Status 404 returned error can't find the container with id fb8b9107606df409fbab749630ca143a0c36ca361f47fd233761267191d1a6ab Jul 30 11:45:00.541548 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.541523 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7"] Jul 30 11:45:00.585409 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.585381 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9"] Jul 30 11:45:00.586442 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:45:00.586416 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2488d182_21ce_4100_b946_d5ab274897b0.slice/crio-e9d9b598f23e376f10871ca5a81bff74db35d61d603d398a06630b09d447c05a WatchSource:0}: Error finding container e9d9b598f23e376f10871ca5a81bff74db35d61d603d398a06630b09d447c05a: Status 404 returned error can't find the container with id e9d9b598f23e376f10871ca5a81bff74db35d61d603d398a06630b09d447c05a Jul 30 11:45:00.710329 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.710304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp"] Jul 30 11:45:00.712478 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:45:00.712451 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958eb994_9b8e_46e4_8238_e27c328e67b0.slice/crio-dfa70d96a91e38c88581ed66b0e7ac857f6c66651c4a798b7b90b895efd76dbb WatchSource:0}: Error finding container dfa70d96a91e38c88581ed66b0e7ac857f6c66651c4a798b7b90b895efd76dbb: Status 404 returned error can't find the container with id dfa70d96a91e38c88581ed66b0e7ac857f6c66651c4a798b7b90b895efd76dbb Jul 30 11:45:00.737488 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.737457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7" event={"ID":"ad425334-0bc9-4ce2-a0fd-02dff8169f61","Type":"ContainerStarted","Data":"fb8b9107606df409fbab749630ca143a0c36ca361f47fd233761267191d1a6ab"} Jul 30 11:45:00.738549 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.738524 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756865-zm68m" event={"ID":"1c074905-a0b3-4abf-aa4c-29effd535a07","Type":"ContainerStarted","Data":"cc04822a655339d575f8277c02ce40379fd5aa98d1730674e575d8d08fa16eae"} Jul 30 11:45:00.739642 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.739624 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7" event={"ID":"383e78c7-f033-4e92-9262-94d1d31ed65d","Type":"ContainerStarted","Data":"9f0c6eb57f357531636da7f7c4fa871de851950d2f46f84696590b25bf0ef010"} Jul 30 11:45:00.740794 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.740772 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9" event={"ID":"2488d182-21ce-4100-b946-d5ab274897b0","Type":"ContainerStarted","Data":"e9d9b598f23e376f10871ca5a81bff74db35d61d603d398a06630b09d447c05a"} Jul 30 11:45:00.741805 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:00.741788 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp" event={"ID":"958eb994-9b8e-46e4-8238-e27c328e67b0","Type":"ContainerStarted","Data":"dfa70d96a91e38c88581ed66b0e7ac857f6c66651c4a798b7b90b895efd76dbb"} Jul 30 11:45:01.751844 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:01.751812 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp" event={"ID":"958eb994-9b8e-46e4-8238-e27c328e67b0","Type":"ContainerStarted","Data":"d8062e51019bb928fd411149feaa6f9fc2db91743fcdaa1b8f869bc4002a8baa"} Jul 30 11:45:01.753464 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:01.753440 2575 generic.go:356] "Generic (PLEG): container finished" podID="ad425334-0bc9-4ce2-a0fd-02dff8169f61" containerID="beef233f79b3a727ee4a8543f41faca5d5cfdf6760a8a06de4824aa9aa6481dc" exitCode=0 Jul 30 11:45:01.753577 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:01.753512 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7" event={"ID":"ad425334-0bc9-4ce2-a0fd-02dff8169f61","Type":"ContainerDied","Data":"beef233f79b3a727ee4a8543f41faca5d5cfdf6760a8a06de4824aa9aa6481dc"} Jul 30 11:45:01.754974 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:01.754956 2575 generic.go:356] "Generic (PLEG): container finished" podID="1c074905-a0b3-4abf-aa4c-29effd535a07" containerID="8031c07b6d405fd858e67f9415dcd52fda29b3e2f99f842bdf5cd1e490ffe9ca" exitCode=0 Jul 30 11:45:01.755083 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:01.755038 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756865-zm68m" event={"ID":"1c074905-a0b3-4abf-aa4c-29effd535a07","Type":"ContainerDied","Data":"8031c07b6d405fd858e67f9415dcd52fda29b3e2f99f842bdf5cd1e490ffe9ca"} Jul 30 11:45:01.756616 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:01.756598 2575 generic.go:356] "Generic (PLEG): container finished" podID="383e78c7-f033-4e92-9262-94d1d31ed65d" containerID="2ff7c6f17af9fdb7cf43817f11844cbfdc462dce8a720ac396ebad1d40d07258" exitCode=0 Jul 30 11:45:01.756707 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:01.756664 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7" event={"ID":"383e78c7-f033-4e92-9262-94d1d31ed65d","Type":"ContainerDied","Data":"2ff7c6f17af9fdb7cf43817f11844cbfdc462dce8a720ac396ebad1d40d07258"} Jul 30 11:45:01.758049 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:01.758031 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9" event={"ID":"2488d182-21ce-4100-b946-d5ab274897b0","Type":"ContainerStarted","Data":"33b353f6578ea18c1621dc5435ba2568babebe32611a152b767505e06a0eff2e"} Jul 30 11:45:01.768363 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:01.768321 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp" podStartSLOduration=1.768308902 podStartE2EDuration="1.768308902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:45:01.765637753 +0000 UTC m=+3409.738498832" watchObservedRunningTime="2026-07-30 11:45:01.768308902 +0000 UTC m=+3409.741169950" Jul 30 11:45:01.793359 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:01.793315 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9" podStartSLOduration=1.793301628 podStartE2EDuration="1.793301628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:45:01.791426238 +0000 UTC m=+3409.764287273" watchObservedRunningTime="2026-07-30 11:45:01.793301628 +0000 UTC m=+3409.766162663" Jul 30 11:45:02.766579 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:02.766543 2575 generic.go:356] "Generic (PLEG): container finished" podID="2488d182-21ce-4100-b946-d5ab274897b0" containerID="33b353f6578ea18c1621dc5435ba2568babebe32611a152b767505e06a0eff2e" exitCode=0 Jul 30 11:45:02.767011 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:02.766637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9" event={"ID":"2488d182-21ce-4100-b946-d5ab274897b0","Type":"ContainerDied","Data":"33b353f6578ea18c1621dc5435ba2568babebe32611a152b767505e06a0eff2e"} Jul 30 11:45:02.768277 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:02.768254 2575 generic.go:356] "Generic (PLEG): container finished" podID="958eb994-9b8e-46e4-8238-e27c328e67b0" containerID="d8062e51019bb928fd411149feaa6f9fc2db91743fcdaa1b8f869bc4002a8baa" exitCode=0 Jul 30 11:45:02.768406 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:02.768286 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp" event={"ID":"958eb994-9b8e-46e4-8238-e27c328e67b0","Type":"ContainerDied","Data":"d8062e51019bb928fd411149feaa6f9fc2db91743fcdaa1b8f869bc4002a8baa"} Jul 30 11:45:02.937541 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:02.937516 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7" Jul 30 11:45:02.990029 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:02.989998 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756865-zm68m" Jul 30 11:45:02.993321 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:02.993298 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7" Jul 30 11:45:03.776214 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.776123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7" event={"ID":"ad425334-0bc9-4ce2-a0fd-02dff8169f61","Type":"ContainerDied","Data":"fb8b9107606df409fbab749630ca143a0c36ca361f47fd233761267191d1a6ab"} Jul 30 11:45:03.776214 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.776158 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7" Jul 30 11:45:03.776663 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.776217 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb8b9107606df409fbab749630ca143a0c36ca361f47fd233761267191d1a6ab" Jul 30 11:45:03.777569 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.777546 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756865-zm68m" Jul 30 11:45:03.777682 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.777588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756865-zm68m" event={"ID":"1c074905-a0b3-4abf-aa4c-29effd535a07","Type":"ContainerDied","Data":"cc04822a655339d575f8277c02ce40379fd5aa98d1730674e575d8d08fa16eae"} Jul 30 11:45:03.777682 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.777634 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc04822a655339d575f8277c02ce40379fd5aa98d1730674e575d8d08fa16eae" Jul 30 11:45:03.778979 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.778955 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7" Jul 30 11:45:03.779073 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.779011 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7" event={"ID":"383e78c7-f033-4e92-9262-94d1d31ed65d","Type":"ContainerDied","Data":"9f0c6eb57f357531636da7f7c4fa871de851950d2f46f84696590b25bf0ef010"} Jul 30 11:45:03.779073 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.779059 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f0c6eb57f357531636da7f7c4fa871de851950d2f46f84696590b25bf0ef010" Jul 30 11:45:03.926873 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.926845 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9" Jul 30 11:45:03.968591 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:03.968566 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp" Jul 30 11:45:04.050554 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:04.050523 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2"] Jul 30 11:45:04.054847 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:04.054815 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756820-j4ks2"] Jul 30 11:45:04.528234 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:04.528148 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e30908c-70a8-4a62-82b5-d7571ea4bc70" path="/var/lib/kubelet/pods/7e30908c-70a8-4a62-82b5-d7571ea4bc70/volumes" Jul 30 11:45:04.786430 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:04.786405 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9" Jul 30 11:45:04.786873 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:04.786447 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9" event={"ID":"2488d182-21ce-4100-b946-d5ab274897b0","Type":"ContainerDied","Data":"e9d9b598f23e376f10871ca5a81bff74db35d61d603d398a06630b09d447c05a"} Jul 30 11:45:04.786873 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:04.786501 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9d9b598f23e376f10871ca5a81bff74db35d61d603d398a06630b09d447c05a" Jul 30 11:45:04.787973 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:04.787954 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp" Jul 30 11:45:04.788109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:04.788022 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp" event={"ID":"958eb994-9b8e-46e4-8238-e27c328e67b0","Type":"ContainerDied","Data":"dfa70d96a91e38c88581ed66b0e7ac857f6c66651c4a798b7b90b895efd76dbb"} Jul 30 11:45:04.788109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:04.788062 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa70d96a91e38c88581ed66b0e7ac857f6c66651c4a798b7b90b895efd76dbb" Jul 30 11:45:10.343192 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:10.343160 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-w2xrv_baba2139-d918-4f6f-9eab-3d14183d9659/payload-processing/0.log" Jul 30 11:45:13.085636 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:13.085606 2575 scope.go:117] "RemoveContainer" containerID="eee7ae94fd3b588fee82b8cfa072f9dbb4992a29743c038c6b166458f1511b48" Jul 30 11:45:29.621591 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.621553 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk"] Jul 30 11:45:29.622033 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622020 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383e78c7-f033-4e92-9262-94d1d31ed65d" containerName="cleanup" Jul 30 11:45:29.622109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622034 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="383e78c7-f033-4e92-9262-94d1d31ed65d" containerName="cleanup" Jul 30 11:45:29.622109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622047 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c074905-a0b3-4abf-aa4c-29effd535a07" containerName="cleanup" Jul 30 11:45:29.622109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622052 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c074905-a0b3-4abf-aa4c-29effd535a07" containerName="cleanup" Jul 30 11:45:29.622109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622060 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad425334-0bc9-4ce2-a0fd-02dff8169f61" containerName="cleanup" Jul 30 11:45:29.622109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622066 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad425334-0bc9-4ce2-a0fd-02dff8169f61" containerName="cleanup" Jul 30 11:45:29.622109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622106 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2488d182-21ce-4100-b946-d5ab274897b0" containerName="cleanup" Jul 30 11:45:29.622109 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622112 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2488d182-21ce-4100-b946-d5ab274897b0" containerName="cleanup" Jul 30 11:45:29.622328 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622118 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958eb994-9b8e-46e4-8238-e27c328e67b0" containerName="cleanup" Jul 30 11:45:29.622328 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622123 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="958eb994-9b8e-46e4-8238-e27c328e67b0" containerName="cleanup" Jul 30 11:45:29.622328 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622187 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c074905-a0b3-4abf-aa4c-29effd535a07" containerName="cleanup" Jul 30 11:45:29.622328 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622196 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="958eb994-9b8e-46e4-8238-e27c328e67b0" containerName="cleanup" Jul 30 11:45:29.622328 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622202 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad425334-0bc9-4ce2-a0fd-02dff8169f61" containerName="cleanup" Jul 30 11:45:29.622328 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622208 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2488d182-21ce-4100-b946-d5ab274897b0" containerName="cleanup" Jul 30 11:45:29.622328 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.622215 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="383e78c7-f033-4e92-9262-94d1d31ed65d" containerName="cleanup" Jul 30 11:45:29.625528 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.625509 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk" Jul 30 11:45:29.627362 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.627337 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:45:29.640606 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.640573 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk"] Jul 30 11:45:29.736000 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.735963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244d4b61-1152-465d-b96c-e967f647083d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk\" (UID: \"244d4b61-1152-465d-b96c-e967f647083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk" Jul 30 11:45:29.837328 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.837280 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244d4b61-1152-465d-b96c-e967f647083d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk\" (UID: \"244d4b61-1152-465d-b96c-e967f647083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk" Jul 30 11:45:29.837894 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.837873 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244d4b61-1152-465d-b96c-e967f647083d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk\" (UID: \"244d4b61-1152-465d-b96c-e967f647083d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk" Jul 30 11:45:29.936674 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:29.936586 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk" Jul 30 11:45:30.285253 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:30.285221 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk"] Jul 30 11:45:30.286266 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:45:30.286242 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244d4b61_1152_465d_b96c_e967f647083d.slice/crio-59859d97c6a617293ce626cba2ceae028a7d7dc422894b123ca87ce7fd6b2250 WatchSource:0}: Error finding container 59859d97c6a617293ce626cba2ceae028a7d7dc422894b123ca87ce7fd6b2250: Status 404 returned error can't find the container with id 59859d97c6a617293ce626cba2ceae028a7d7dc422894b123ca87ce7fd6b2250 Jul 30 11:45:30.968778 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:30.968745 2575 generic.go:356] "Generic (PLEG): container finished" podID="244d4b61-1152-465d-b96c-e967f647083d" containerID="77ebb1079023d06f61472046a02c781be8fa2321835067ae817684affe6654b1" exitCode=0 Jul 30 11:45:30.969182 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:30.968822 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk" event={"ID":"244d4b61-1152-465d-b96c-e967f647083d","Type":"ContainerDied","Data":"77ebb1079023d06f61472046a02c781be8fa2321835067ae817684affe6654b1"} Jul 30 11:45:30.969182 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:30.968873 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk" event={"ID":"244d4b61-1152-465d-b96c-e967f647083d","Type":"ContainerStarted","Data":"59859d97c6a617293ce626cba2ceae028a7d7dc422894b123ca87ce7fd6b2250"} Jul 30 11:45:32.120177 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:32.120150 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk" Jul 30 11:45:32.160442 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:32.160411 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244d4b61-1152-465d-b96c-e967f647083d-maas-api-service-ca\") pod \"244d4b61-1152-465d-b96c-e967f647083d\" (UID: \"244d4b61-1152-465d-b96c-e967f647083d\") " Jul 30 11:45:32.160830 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:32.160804 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244d4b61-1152-465d-b96c-e967f647083d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "244d4b61-1152-465d-b96c-e967f647083d" (UID: "244d4b61-1152-465d-b96c-e967f647083d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:45:32.261346 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:32.261261 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/244d4b61-1152-465d-b96c-e967f647083d-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:45:32.984097 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:32.983999 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk" event={"ID":"244d4b61-1152-465d-b96c-e967f647083d","Type":"ContainerDied","Data":"59859d97c6a617293ce626cba2ceae028a7d7dc422894b123ca87ce7fd6b2250"} Jul 30 11:45:32.984097 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:32.984047 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59859d97c6a617293ce626cba2ceae028a7d7dc422894b123ca87ce7fd6b2250" Jul 30 11:45:32.984097 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:32.984010 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk" Jul 30 11:45:39.088339 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:39.088281 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk"] Jul 30 11:45:39.093558 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:39.093534 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a93cc2-j8ggk"] Jul 30 11:45:40.528281 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:40.528248 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="244d4b61-1152-465d-b96c-e967f647083d" path="/var/lib/kubelet/pods/244d4b61-1152-465d-b96c-e967f647083d/volumes" Jul 30 11:45:50.302680 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.302639 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn"] Jul 30 11:45:50.303323 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.303304 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="244d4b61-1152-465d-b96c-e967f647083d" containerName="revoke-keys" Jul 30 11:45:50.303323 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.303324 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="244d4b61-1152-465d-b96c-e967f647083d" containerName="revoke-keys" Jul 30 11:45:50.303483 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.303437 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="244d4b61-1152-465d-b96c-e967f647083d" containerName="revoke-keys" Jul 30 11:45:50.307438 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.307419 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn" Jul 30 11:45:50.309604 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.309582 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:45:50.314636 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.314613 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn"] Jul 30 11:45:50.447500 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.447459 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34d1a90-56f7-4850-80c1-de95d738c108-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn\" (UID: \"c34d1a90-56f7-4850-80c1-de95d738c108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn" Jul 30 11:45:50.548726 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.548692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34d1a90-56f7-4850-80c1-de95d738c108-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn\" (UID: \"c34d1a90-56f7-4850-80c1-de95d738c108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn" Jul 30 11:45:50.549345 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.549324 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34d1a90-56f7-4850-80c1-de95d738c108-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn\" (UID: \"c34d1a90-56f7-4850-80c1-de95d738c108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn" Jul 30 11:45:50.619777 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.619694 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn" Jul 30 11:45:50.979461 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:50.979434 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn"] Jul 30 11:45:50.980889 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:45:50.980855 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34d1a90_56f7_4850_80c1_de95d738c108.slice/crio-7c0bfa47c3d1922a08272454c3aa96ee41b2f310c06eca098e6bb8878a1f9849 WatchSource:0}: Error finding container 7c0bfa47c3d1922a08272454c3aa96ee41b2f310c06eca098e6bb8878a1f9849: Status 404 returned error can't find the container with id 7c0bfa47c3d1922a08272454c3aa96ee41b2f310c06eca098e6bb8878a1f9849 Jul 30 11:45:51.112297 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:51.112260 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn" event={"ID":"c34d1a90-56f7-4850-80c1-de95d738c108","Type":"ContainerStarted","Data":"7c0bfa47c3d1922a08272454c3aa96ee41b2f310c06eca098e6bb8878a1f9849"} Jul 30 11:45:52.120699 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:52.120666 2575 generic.go:356] "Generic (PLEG): container finished" podID="c34d1a90-56f7-4850-80c1-de95d738c108" containerID="d697652c921f681f380d89787d1e1ad315d7cfe8bbd0ffe8902c5413074e7ffc" exitCode=0 Jul 30 11:45:52.121130 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:52.120755 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn" event={"ID":"c34d1a90-56f7-4850-80c1-de95d738c108","Type":"ContainerDied","Data":"d697652c921f681f380d89787d1e1ad315d7cfe8bbd0ffe8902c5413074e7ffc"} Jul 30 11:45:53.272469 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:53.272445 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn" Jul 30 11:45:53.376446 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:53.376415 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34d1a90-56f7-4850-80c1-de95d738c108-maas-api-service-ca\") pod \"c34d1a90-56f7-4850-80c1-de95d738c108\" (UID: \"c34d1a90-56f7-4850-80c1-de95d738c108\") " Jul 30 11:45:53.376746 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:53.376723 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34d1a90-56f7-4850-80c1-de95d738c108-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c34d1a90-56f7-4850-80c1-de95d738c108" (UID: "c34d1a90-56f7-4850-80c1-de95d738c108"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:45:53.477717 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:53.477636 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34d1a90-56f7-4850-80c1-de95d738c108-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:45:54.136300 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:54.136272 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn" Jul 30 11:45:54.136510 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:54.136271 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn" event={"ID":"c34d1a90-56f7-4850-80c1-de95d738c108","Type":"ContainerDied","Data":"7c0bfa47c3d1922a08272454c3aa96ee41b2f310c06eca098e6bb8878a1f9849"} Jul 30 11:45:54.136510 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:45:54.136396 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c0bfa47c3d1922a08272454c3aa96ee41b2f310c06eca098e6bb8878a1f9849" Jul 30 11:46:07.370536 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:46:07.370501 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7"] Jul 30 11:46:07.373466 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:46:07.373441 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-d722d1-29756865-hd9x7"] Jul 30 11:46:07.585660 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:46:07.585618 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn"] Jul 30 11:46:07.588521 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:46:07.588492 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d722d1-qnbsn"] Jul 30 11:46:08.528148 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:46:08.528102 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383e78c7-f033-4e92-9262-94d1d31ed65d" path="/var/lib/kubelet/pods/383e78c7-f033-4e92-9262-94d1d31ed65d/volumes" Jul 30 11:46:08.528542 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:46:08.528441 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c34d1a90-56f7-4850-80c1-de95d738c108" path="/var/lib/kubelet/pods/c34d1a90-56f7-4850-80c1-de95d738c108/volumes" Jul 30 11:47:05.500483 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.500452 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z"] Jul 30 11:47:05.501047 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.500703 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" podUID="51914b67-c18f-476a-9f34-d598210e5ee0" containerName="maas-api" containerID="cri-o://1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48" gracePeriod=30 Jul 30 11:47:05.521438 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.521408 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp"] Jul 30 11:47:05.528794 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.528763 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-f0ce38-29756865-287dp"] Jul 30 11:47:05.770108 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.770067 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:47:05.898317 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.898285 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7cg9\" (UniqueName: \"kubernetes.io/projected/51914b67-c18f-476a-9f34-d598210e5ee0-kube-api-access-d7cg9\") pod \"51914b67-c18f-476a-9f34-d598210e5ee0\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " Jul 30 11:47:05.898518 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.898399 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51914b67-c18f-476a-9f34-d598210e5ee0-openshift-service-ca\") pod \"51914b67-c18f-476a-9f34-d598210e5ee0\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " Jul 30 11:47:05.898518 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.898438 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51914b67-c18f-476a-9f34-d598210e5ee0-maas-api-tls\") pod \"51914b67-c18f-476a-9f34-d598210e5ee0\" (UID: \"51914b67-c18f-476a-9f34-d598210e5ee0\") " Jul 30 11:47:05.898846 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.898806 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51914b67-c18f-476a-9f34-d598210e5ee0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "51914b67-c18f-476a-9f34-d598210e5ee0" (UID: "51914b67-c18f-476a-9f34-d598210e5ee0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:47:05.900540 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.900518 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51914b67-c18f-476a-9f34-d598210e5ee0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "51914b67-c18f-476a-9f34-d598210e5ee0" (UID: "51914b67-c18f-476a-9f34-d598210e5ee0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 11:47:05.900641 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.900517 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51914b67-c18f-476a-9f34-d598210e5ee0-kube-api-access-d7cg9" (OuterVolumeSpecName: "kube-api-access-d7cg9") pod "51914b67-c18f-476a-9f34-d598210e5ee0" (UID: "51914b67-c18f-476a-9f34-d598210e5ee0"). InnerVolumeSpecName "kube-api-access-d7cg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:47:05.999569 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.999538 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51914b67-c18f-476a-9f34-d598210e5ee0-openshift-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:05.999569 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.999563 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51914b67-c18f-476a-9f34-d598210e5ee0-maas-api-tls\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:05.999569 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:05.999572 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7cg9\" (UniqueName: \"kubernetes.io/projected/51914b67-c18f-476a-9f34-d598210e5ee0-kube-api-access-d7cg9\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:06.528279 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:06.528238 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="958eb994-9b8e-46e4-8238-e27c328e67b0" path="/var/lib/kubelet/pods/958eb994-9b8e-46e4-8238-e27c328e67b0/volumes" Jul 30 11:47:06.604074 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:06.604036 2575 generic.go:356] "Generic (PLEG): container finished" podID="51914b67-c18f-476a-9f34-d598210e5ee0" containerID="1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48" exitCode=0 Jul 30 11:47:06.604259 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:06.604116 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" Jul 30 11:47:06.604259 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:06.604135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" event={"ID":"51914b67-c18f-476a-9f34-d598210e5ee0","Type":"ContainerDied","Data":"1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48"} Jul 30 11:47:06.604259 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:06.604222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z" event={"ID":"51914b67-c18f-476a-9f34-d598210e5ee0","Type":"ContainerDied","Data":"8ac3fba759c4892ef8e3dc4d20403d5a0f41a4044b593ce3aa108834f9a29013"} Jul 30 11:47:06.604259 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:06.604245 2575 scope.go:117] "RemoveContainer" containerID="1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48" Jul 30 11:47:06.617608 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:06.617560 2575 scope.go:117] "RemoveContainer" containerID="1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48" Jul 30 11:47:06.617936 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:47:06.617916 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48\": container with ID starting with 1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48 not found: ID does not exist" containerID="1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48" Jul 30 11:47:06.617989 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:06.617946 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48"} err="failed to get container status \"1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48\": rpc error: code = NotFound desc = could not find container \"1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48\": container with ID starting with 1e18f327ea5b246a5cdf3167d6b2f90386be17b6fc2e88e6923dfa9e975c3d48 not found: ID does not exist" Jul 30 11:47:06.623816 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:06.623792 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z"] Jul 30 11:47:06.627976 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:06.627954 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f0ce38-6d6fcb87f6-bhc4z"] Jul 30 11:47:08.529610 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:08.529574 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51914b67-c18f-476a-9f34-d598210e5ee0" path="/var/lib/kubelet/pods/51914b67-c18f-476a-9f34-d598210e5ee0/volumes" Jul 30 11:47:12.836392 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:12.836357 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52"] Jul 30 11:47:12.836761 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:12.836682 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" containerID="cri-o://df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48" gracePeriod=30 Jul 30 11:47:18.111457 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.111397 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:47:18.117138 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117113 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-envoy\") pod \"21c68677-723b-4d0c-90b6-6ff222e192f5\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " Jul 30 11:47:18.117263 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117155 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c68677-723b-4d0c-90b6-6ff222e192f5-istiod-ca-cert\") pod \"21c68677-723b-4d0c-90b6-6ff222e192f5\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " Jul 30 11:47:18.117263 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117189 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-credential-socket\") pod \"21c68677-723b-4d0c-90b6-6ff222e192f5\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " Jul 30 11:47:18.117359 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117271 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-token\") pod \"21c68677-723b-4d0c-90b6-6ff222e192f5\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " Jul 30 11:47:18.117359 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117306 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-podinfo\") pod \"21c68677-723b-4d0c-90b6-6ff222e192f5\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " Jul 30 11:47:18.117359 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117347 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-certs\") pod \"21c68677-723b-4d0c-90b6-6ff222e192f5\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " Jul 30 11:47:18.117516 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117384 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwt5w\" (UniqueName: \"kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-kube-api-access-gwt5w\") pod \"21c68677-723b-4d0c-90b6-6ff222e192f5\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " Jul 30 11:47:18.117516 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117419 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-data\") pod \"21c68677-723b-4d0c-90b6-6ff222e192f5\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " Jul 30 11:47:18.117516 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117459 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-socket\") pod \"21c68677-723b-4d0c-90b6-6ff222e192f5\" (UID: \"21c68677-723b-4d0c-90b6-6ff222e192f5\") " Jul 30 11:47:18.117676 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117626 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c68677-723b-4d0c-90b6-6ff222e192f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21c68677-723b-4d0c-90b6-6ff222e192f5" (UID: "21c68677-723b-4d0c-90b6-6ff222e192f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:47:18.117882 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117808 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21c68677-723b-4d0c-90b6-6ff222e192f5" (UID: "21c68677-723b-4d0c-90b6-6ff222e192f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:47:18.117882 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117843 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21c68677-723b-4d0c-90b6-6ff222e192f5-istiod-ca-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:18.118070 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117933 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21c68677-723b-4d0c-90b6-6ff222e192f5" (UID: "21c68677-723b-4d0c-90b6-6ff222e192f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:47:18.118070 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.117954 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21c68677-723b-4d0c-90b6-6ff222e192f5" (UID: "21c68677-723b-4d0c-90b6-6ff222e192f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:47:18.118070 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.118033 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "21c68677-723b-4d0c-90b6-6ff222e192f5" (UID: "21c68677-723b-4d0c-90b6-6ff222e192f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:47:18.119471 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.119449 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21c68677-723b-4d0c-90b6-6ff222e192f5" (UID: "21c68677-723b-4d0c-90b6-6ff222e192f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:47:18.119723 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.119708 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21c68677-723b-4d0c-90b6-6ff222e192f5" (UID: "21c68677-723b-4d0c-90b6-6ff222e192f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 11:47:18.119880 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.119861 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-kube-api-access-gwt5w" (OuterVolumeSpecName: "kube-api-access-gwt5w") pod "21c68677-723b-4d0c-90b6-6ff222e192f5" (UID: "21c68677-723b-4d0c-90b6-6ff222e192f5"). InnerVolumeSpecName "kube-api-access-gwt5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:47:18.119880 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.119872 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "21c68677-723b-4d0c-90b6-6ff222e192f5" (UID: "21c68677-723b-4d0c-90b6-6ff222e192f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:47:18.218502 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.218471 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-token\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:18.218502 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.218499 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-podinfo\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:18.218720 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.218512 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-certs\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:18.218720 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.218524 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwt5w\" (UniqueName: \"kubernetes.io/projected/21c68677-723b-4d0c-90b6-6ff222e192f5-kube-api-access-gwt5w\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:18.218720 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.218533 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-data\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:18.218720 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.218541 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-workload-socket\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:18.218720 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.218548 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-istio-envoy\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:18.218720 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.218559 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21c68677-723b-4d0c-90b6-6ff222e192f5-credential-socket\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:18.684293 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.684261 2575 generic.go:356] "Generic (PLEG): container finished" podID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerID="df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48" exitCode=0 Jul 30 11:47:18.684490 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.684328 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" Jul 30 11:47:18.684490 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.684355 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" event={"ID":"21c68677-723b-4d0c-90b6-6ff222e192f5","Type":"ContainerDied","Data":"df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48"} Jul 30 11:47:18.684490 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.684424 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52" event={"ID":"21c68677-723b-4d0c-90b6-6ff222e192f5","Type":"ContainerDied","Data":"911a070bb5f5b0ab70e43b37d6012f2f706eb9b2666549140fbd2862ecb6d792"} Jul 30 11:47:18.684490 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.684454 2575 scope.go:117] "RemoveContainer" containerID="df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48" Jul 30 11:47:18.694444 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.694422 2575 scope.go:117] "RemoveContainer" containerID="f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf" Jul 30 11:47:18.701493 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.701470 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52"] Jul 30 11:47:18.704682 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.704663 2575 scope.go:117] "RemoveContainer" containerID="df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48" Jul 30 11:47:18.704758 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.704698 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f0ce38-openshift-default-88b75c76b-xwt52"] Jul 30 11:47:18.704953 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:47:18.704926 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48\": container with ID starting with df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48 not found: ID does not exist" containerID="df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48" Jul 30 11:47:18.705004 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.704960 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48"} err="failed to get container status \"df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48\": rpc error: code = NotFound desc = could not find container \"df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48\": container with ID starting with df7ca7740c58e841caaad671665a727f60b47c58b66209d3c48c5d9804308e48 not found: ID does not exist" Jul 30 11:47:18.705004 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.704978 2575 scope.go:117] "RemoveContainer" containerID="f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf" Jul 30 11:47:18.705245 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:47:18.705226 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf\": container with ID starting with f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf not found: ID does not exist" containerID="f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf" Jul 30 11:47:18.705318 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:18.705250 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf"} err="failed to get container status \"f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf\": rpc error: code = NotFound desc = could not find container \"f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf\": container with ID starting with f811b7e3f32edc6816adc0a00ebebdc16db7ebd7d3b59bc4ef4a34d4f65618cf not found: ID does not exist" Jul 30 11:47:20.302488 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:20.302455 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7"] Jul 30 11:47:20.305000 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:20.304974 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-060e9f-29756865-d6qg7"] Jul 30 11:47:20.529910 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:20.529873 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" path="/var/lib/kubelet/pods/21c68677-723b-4d0c-90b6-6ff222e192f5/volumes" Jul 30 11:47:20.530656 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:20.530620 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad425334-0bc9-4ce2-a0fd-02dff8169f61" path="/var/lib/kubelet/pods/ad425334-0bc9-4ce2-a0fd-02dff8169f61/volumes" Jul 30 11:47:28.245750 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.245711 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22"] Jul 30 11:47:28.246146 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.245962 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" containerID="cri-o://cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23" gracePeriod=30 Jul 30 11:47:28.684824 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.684793 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw"] Jul 30 11:47:28.685259 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685247 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51914b67-c18f-476a-9f34-d598210e5ee0" containerName="maas-api" Jul 30 11:47:28.685314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685261 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="51914b67-c18f-476a-9f34-d598210e5ee0" containerName="maas-api" Jul 30 11:47:28.685314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685280 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" Jul 30 11:47:28.685314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685286 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" Jul 30 11:47:28.685314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685295 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" Jul 30 11:47:28.685314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685300 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" Jul 30 11:47:28.685314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685310 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34d1a90-56f7-4850-80c1-de95d738c108" containerName="revoke-keys" Jul 30 11:47:28.685314 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685315 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34d1a90-56f7-4850-80c1-de95d738c108" containerName="revoke-keys" Jul 30 11:47:28.685534 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685383 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" Jul 30 11:47:28.685534 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685391 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="21c68677-723b-4d0c-90b6-6ff222e192f5" containerName="istio-proxy" Jul 30 11:47:28.685534 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685397 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c34d1a90-56f7-4850-80c1-de95d738c108" containerName="revoke-keys" Jul 30 11:47:28.685534 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.685406 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="51914b67-c18f-476a-9f34-d598210e5ee0" containerName="maas-api" Jul 30 11:47:28.688017 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.688002 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" Jul 30 11:47:28.690354 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.690333 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ztd2n\"" Jul 30 11:47:28.697803 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.697781 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw"] Jul 30 11:47:28.724967 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.724930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a8de82-429b-4d72-b70d-c41125ff11df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw\" (UID: \"58a8de82-429b-4d72-b70d-c41125ff11df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" Jul 30 11:47:28.826426 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.826393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a8de82-429b-4d72-b70d-c41125ff11df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw\" (UID: \"58a8de82-429b-4d72-b70d-c41125ff11df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" Jul 30 11:47:28.827044 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.827021 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a8de82-429b-4d72-b70d-c41125ff11df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw\" (UID: \"58a8de82-429b-4d72-b70d-c41125ff11df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" Jul 30 11:47:28.999977 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:28.999897 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" Jul 30 11:47:29.141032 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:47:29.140998 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a8de82_429b_4d72_b70d_c41125ff11df.slice/crio-e9b8eb67bdaeb739f1e88516538ed0550527add86a2587606c1e9bb12481036f WatchSource:0}: Error finding container e9b8eb67bdaeb739f1e88516538ed0550527add86a2587606c1e9bb12481036f: Status 404 returned error can't find the container with id e9b8eb67bdaeb739f1e88516538ed0550527add86a2587606c1e9bb12481036f Jul 30 11:47:29.142337 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:29.141661 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw"] Jul 30 11:47:29.755939 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:29.755839 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" event={"ID":"58a8de82-429b-4d72-b70d-c41125ff11df","Type":"ContainerStarted","Data":"09cc53a9a7e9032dd64e26974ffed86128a8b7fe3c3f86e5b415899912cb8886"} Jul 30 11:47:29.755939 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:29.755904 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" event={"ID":"58a8de82-429b-4d72-b70d-c41125ff11df","Type":"ContainerStarted","Data":"e9b8eb67bdaeb739f1e88516538ed0550527add86a2587606c1e9bb12481036f"} Jul 30 11:47:29.756676 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:29.756656 2575 patch_prober.go:28] interesting pod/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 11:47:29.756732 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:29.756706 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 11:47:29.774030 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:29.773975 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" podStartSLOduration=1.773960281 podStartE2EDuration="1.773960281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 11:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:47:29.770953746 +0000 UTC m=+3557.743814787" watchObservedRunningTime="2026-07-30 11:47:29.773960281 +0000 UTC m=+3557.746821315" Jul 30 11:47:30.764267 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:30.764227 2575 generic.go:356] "Generic (PLEG): container finished" podID="58a8de82-429b-4d72-b70d-c41125ff11df" containerID="09cc53a9a7e9032dd64e26974ffed86128a8b7fe3c3f86e5b415899912cb8886" exitCode=0 Jul 30 11:47:30.764694 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:30.764317 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" event={"ID":"58a8de82-429b-4d72-b70d-c41125ff11df","Type":"ContainerDied","Data":"09cc53a9a7e9032dd64e26974ffed86128a8b7fe3c3f86e5b415899912cb8886"} Jul 30 11:47:31.906331 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:31.906307 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" Jul 30 11:47:31.958154 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:31.958124 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a8de82-429b-4d72-b70d-c41125ff11df-maas-api-service-ca\") pod \"58a8de82-429b-4d72-b70d-c41125ff11df\" (UID: \"58a8de82-429b-4d72-b70d-c41125ff11df\") " Jul 30 11:47:31.958483 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:31.958459 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a8de82-429b-4d72-b70d-c41125ff11df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58a8de82-429b-4d72-b70d-c41125ff11df" (UID: "58a8de82-429b-4d72-b70d-c41125ff11df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:47:32.059208 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:32.059180 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a8de82-429b-4d72-b70d-c41125ff11df-maas-api-service-ca\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:32.777860 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:32.777828 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" Jul 30 11:47:32.778038 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:32.777833 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw" event={"ID":"58a8de82-429b-4d72-b70d-c41125ff11df","Type":"ContainerDied","Data":"e9b8eb67bdaeb739f1e88516538ed0550527add86a2587606c1e9bb12481036f"} Jul 30 11:47:32.778038 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:32.777947 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9b8eb67bdaeb739f1e88516538ed0550527add86a2587606c1e9bb12481036f" Jul 30 11:47:33.507737 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.507713 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:47:33.572342 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572302 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922c1710-1d7b-475a-b72d-708e7414cd49-istio-podinfo\") pod \"922c1710-1d7b-475a-b72d-708e7414cd49\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " Jul 30 11:47:33.572546 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572380 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-envoy\") pod \"922c1710-1d7b-475a-b72d-708e7414cd49\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " Jul 30 11:47:33.572546 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572410 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922c1710-1d7b-475a-b72d-708e7414cd49-istiod-ca-cert\") pod \"922c1710-1d7b-475a-b72d-708e7414cd49\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " Jul 30 11:47:33.572546 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572434 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-istio-token\") pod \"922c1710-1d7b-475a-b72d-708e7414cd49\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " Jul 30 11:47:33.572546 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572458 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-socket\") pod \"922c1710-1d7b-475a-b72d-708e7414cd49\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " Jul 30 11:47:33.572546 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572483 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-credential-socket\") pod \"922c1710-1d7b-475a-b72d-708e7414cd49\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " Jul 30 11:47:33.572776 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572637 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-data\") pod \"922c1710-1d7b-475a-b72d-708e7414cd49\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " Jul 30 11:47:33.572776 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572707 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-certs\") pod \"922c1710-1d7b-475a-b72d-708e7414cd49\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " Jul 30 11:47:33.572852 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572773 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hqjs\" (UniqueName: \"kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-kube-api-access-5hqjs\") pod \"922c1710-1d7b-475a-b72d-708e7414cd49\" (UID: \"922c1710-1d7b-475a-b72d-708e7414cd49\") " Jul 30 11:47:33.572852 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572800 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "922c1710-1d7b-475a-b72d-708e7414cd49" (UID: "922c1710-1d7b-475a-b72d-708e7414cd49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:47:33.572941 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572879 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922c1710-1d7b-475a-b72d-708e7414cd49-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "922c1710-1d7b-475a-b72d-708e7414cd49" (UID: "922c1710-1d7b-475a-b72d-708e7414cd49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:47:33.572941 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.572901 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-data" (OuterVolumeSpecName: "istio-data") pod "922c1710-1d7b-475a-b72d-708e7414cd49" (UID: "922c1710-1d7b-475a-b72d-708e7414cd49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:47:33.573021 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.573007 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "922c1710-1d7b-475a-b72d-708e7414cd49" (UID: "922c1710-1d7b-475a-b72d-708e7414cd49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:47:33.573211 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.573186 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922c1710-1d7b-475a-b72d-708e7414cd49-istiod-ca-cert\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:33.573211 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.573212 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-socket\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:33.573451 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.573224 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-data\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:33.573451 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.573234 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-workload-certs\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:33.573451 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.573322 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "922c1710-1d7b-475a-b72d-708e7414cd49" (UID: "922c1710-1d7b-475a-b72d-708e7414cd49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:47:33.574760 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.574739 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-istio-token" (OuterVolumeSpecName: "istio-token") pod "922c1710-1d7b-475a-b72d-708e7414cd49" (UID: "922c1710-1d7b-475a-b72d-708e7414cd49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:47:33.574849 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.574824 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-kube-api-access-5hqjs" (OuterVolumeSpecName: "kube-api-access-5hqjs") pod "922c1710-1d7b-475a-b72d-708e7414cd49" (UID: "922c1710-1d7b-475a-b72d-708e7414cd49"). InnerVolumeSpecName "kube-api-access-5hqjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:47:33.574984 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.574967 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922c1710-1d7b-475a-b72d-708e7414cd49-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "922c1710-1d7b-475a-b72d-708e7414cd49" (UID: "922c1710-1d7b-475a-b72d-708e7414cd49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 30 11:47:33.575110 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.575079 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "922c1710-1d7b-475a-b72d-708e7414cd49" (UID: "922c1710-1d7b-475a-b72d-708e7414cd49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 11:47:33.674179 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.674144 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hqjs\" (UniqueName: \"kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-kube-api-access-5hqjs\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:33.674179 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.674177 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922c1710-1d7b-475a-b72d-708e7414cd49-istio-podinfo\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:33.674385 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.674190 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-istio-envoy\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:33.674385 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.674203 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922c1710-1d7b-475a-b72d-708e7414cd49-istio-token\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:33.674385 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.674214 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922c1710-1d7b-475a-b72d-708e7414cd49-credential-socket\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:33.785186 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.785149 2575 generic.go:356] "Generic (PLEG): container finished" podID="922c1710-1d7b-475a-b72d-708e7414cd49" containerID="cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23" exitCode=0 Jul 30 11:47:33.785186 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.785184 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" Jul 30 11:47:33.785449 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.785201 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" event={"ID":"922c1710-1d7b-475a-b72d-708e7414cd49","Type":"ContainerDied","Data":"cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23"} Jul 30 11:47:33.785449 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.785265 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22" event={"ID":"922c1710-1d7b-475a-b72d-708e7414cd49","Type":"ContainerDied","Data":"c982e53eb556033959d5a558e8003c0e9bface38e5e50282bce71c64d8352e25"} Jul 30 11:47:33.785449 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.785298 2575 scope.go:117] "RemoveContainer" containerID="cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23" Jul 30 11:47:33.796039 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.796021 2575 scope.go:117] "RemoveContainer" containerID="ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc" Jul 30 11:47:33.804626 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.804602 2575 scope.go:117] "RemoveContainer" containerID="cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23" Jul 30 11:47:33.804956 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:47:33.804902 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23\": container with ID starting with cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23 not found: ID does not exist" containerID="cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23" Jul 30 11:47:33.804956 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.804931 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23"} err="failed to get container status \"cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23\": rpc error: code = NotFound desc = could not find container \"cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23\": container with ID starting with cc0442d513354d9e3813bf4a91d3e9f72d9f2c4e23ed1a689985a0a14ac59d23 not found: ID does not exist" Jul 30 11:47:33.805277 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.804963 2575 scope.go:117] "RemoveContainer" containerID="ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc" Jul 30 11:47:33.805334 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:47:33.805308 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc\": container with ID starting with ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc not found: ID does not exist" containerID="ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc" Jul 30 11:47:33.805382 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.805357 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc"} err="failed to get container status \"ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc\": rpc error: code = NotFound desc = could not find container \"ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc\": container with ID starting with ae98d5d89d671a918165921a9e11a54cf278d638f27a93b744eaf74951711ddc not found: ID does not exist" Jul 30 11:47:33.808430 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.808407 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22"] Jul 30 11:47:33.811847 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:33.811827 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-060e9f-openshift-default-765f5f4fc6-lwg22"] Jul 30 11:47:34.528291 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:34.528258 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" path="/var/lib/kubelet/pods/922c1710-1d7b-475a-b72d-708e7414cd49/volumes" Jul 30 11:47:35.129416 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.129379 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9"] Jul 30 11:47:35.133643 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.133616 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-945db9-29756865-4kkv9"] Jul 30 11:47:35.184106 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.184042 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk"] Jul 30 11:47:35.184556 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.184360 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" podUID="83631c3a-ce95-4120-ae91-3004c098c354" containerName="payload-pre-processing" containerID="cri-o://ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0" gracePeriod=30 Jul 30 11:47:35.466210 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.466181 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:47:35.518256 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.518219 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw"] Jul 30 11:47:35.522139 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.522111 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-945db9-2bqtw"] Jul 30 11:47:35.590793 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.590759 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j66t9\" (UniqueName: \"kubernetes.io/projected/83631c3a-ce95-4120-ae91-3004c098c354-kube-api-access-j66t9\") pod \"83631c3a-ce95-4120-ae91-3004c098c354\" (UID: \"83631c3a-ce95-4120-ae91-3004c098c354\") " Jul 30 11:47:35.591222 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.590854 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83631c3a-ce95-4120-ae91-3004c098c354-plugins-config-volume\") pod \"83631c3a-ce95-4120-ae91-3004c098c354\" (UID: \"83631c3a-ce95-4120-ae91-3004c098c354\") " Jul 30 11:47:35.591222 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.591200 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83631c3a-ce95-4120-ae91-3004c098c354-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "83631c3a-ce95-4120-ae91-3004c098c354" (UID: "83631c3a-ce95-4120-ae91-3004c098c354"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 11:47:35.591328 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.591282 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83631c3a-ce95-4120-ae91-3004c098c354-plugins-config-volume\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:35.593057 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.593035 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83631c3a-ce95-4120-ae91-3004c098c354-kube-api-access-j66t9" (OuterVolumeSpecName: "kube-api-access-j66t9") pod "83631c3a-ce95-4120-ae91-3004c098c354" (UID: "83631c3a-ce95-4120-ae91-3004c098c354"). InnerVolumeSpecName "kube-api-access-j66t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 11:47:35.692600 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.692522 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j66t9\" (UniqueName: \"kubernetes.io/projected/83631c3a-ce95-4120-ae91-3004c098c354-kube-api-access-j66t9\") on node \"ip-10-0-138-131.ec2.internal\" DevicePath \"\"" Jul 30 11:47:35.800356 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.800310 2575 generic.go:356] "Generic (PLEG): container finished" podID="83631c3a-ce95-4120-ae91-3004c098c354" containerID="ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0" exitCode=0 Jul 30 11:47:35.800619 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.800377 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" Jul 30 11:47:35.800619 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.800478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" event={"ID":"83631c3a-ce95-4120-ae91-3004c098c354","Type":"ContainerDied","Data":"ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0"} Jul 30 11:47:35.800619 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.800529 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk" event={"ID":"83631c3a-ce95-4120-ae91-3004c098c354","Type":"ContainerDied","Data":"3f76178198a0ac55e4bb4d2d6723c8c6a03b9670916edfe008033e04b8752b09"} Jul 30 11:47:35.800619 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.800556 2575 scope.go:117] "RemoveContainer" containerID="ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0" Jul 30 11:47:35.809864 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.809847 2575 scope.go:117] "RemoveContainer" containerID="ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0" Jul 30 11:47:35.810137 ip-10-0-138-131 kubenswrapper[2575]: E0730 11:47:35.810117 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0\": container with ID starting with ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0 not found: ID does not exist" containerID="ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0" Jul 30 11:47:35.810213 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.810145 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0"} err="failed to get container status \"ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0\": rpc error: code = NotFound desc = could not find container \"ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0\": container with ID starting with ce39eb2f294594f942d19aa0862b14d7e7ee713fe1d4690928c371bfc85085e0 not found: ID does not exist" Jul 30 11:47:35.821893 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.821868 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk"] Jul 30 11:47:35.829962 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:35.828747 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-945db9-74889db98d-588dk"] Jul 30 11:47:36.529067 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:36.529036 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2488d182-21ce-4100-b946-d5ab274897b0" path="/var/lib/kubelet/pods/2488d182-21ce-4100-b946-d5ab274897b0/volumes" Jul 30 11:47:36.529352 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:36.529339 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58a8de82-429b-4d72-b70d-c41125ff11df" path="/var/lib/kubelet/pods/58a8de82-429b-4d72-b70d-c41125ff11df/volumes" Jul 30 11:47:36.529647 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:36.529636 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83631c3a-ce95-4120-ae91-3004c098c354" path="/var/lib/kubelet/pods/83631c3a-ce95-4120-ae91-3004c098c354/volumes" Jul 30 11:47:48.759448 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:48.759415 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d558b5f66-q6g9n_b3ec4cdd-0fa1-4009-914d-5f4b5e119024/manager/0.log" Jul 30 11:47:49.002963 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:49.002932 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7f7f8_5241e5b6-69c3-42e2-b691-3ead22d1e54a/manager/0.log" Jul 30 11:47:49.405171 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:49.405145 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d97895fdd-r8wjr_cebd5340-08de-4b45-aae8-01a1ee99ab76/copy-manifests/0.log" Jul 30 11:47:49.412665 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:49.412641 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d97895fdd-r8wjr_cebd5340-08de-4b45-aae8-01a1ee99ab76/manager/0.log" Jul 30 11:47:49.746601 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:49.746490 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-669686cd4-rhjj8_d2f549d3-8110-42c6-ae80-d46a2c102a62/maas-api/0.log" Jul 30 11:47:49.858040 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:49.858005 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756835-wl69b_04665bef-eab4-49d5-b506-0298470c6382/cleanup/0.log" Jul 30 11:47:50.099301 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:50.099273 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756865-zm68m_1c074905-a0b3-4abf-aa4c-29effd535a07/cleanup/0.log" Jul 30 11:47:50.212139 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:50.212105 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jtf5w_fe27bc51-0b91-4df6-b459-cdb6e93b4873/postgres/0.log" Jul 30 11:47:51.254055 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:51.254023 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-rk8vx_bcbcd504-d10a-4db5-b254-05f7e39ed612/manager/0.log" Jul 30 11:47:51.482997 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:51.482968 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-mk2vz_ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755/manager/0.log" Jul 30 11:47:51.725411 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:51.725366 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-zqmss_45a7ed45-e7c6-4faa-b60b-1f2c62584111/manager/0.log" Jul 30 11:47:52.201972 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:52.201941 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qtsjv_df10590c-6edd-48b3-a156-bf21353dacd8/discovery/0.log" Jul 30 11:47:52.443449 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:52.443397 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8674569cb6-nlpn8_4fe82e77-f51a-452c-9213-0e779ce306ca/kube-auth-proxy/0.log" Jul 30 11:47:52.710416 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:52.710379 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-h8q2d_2d7944ed-a849-430c-b6eb-bb8bd57754c0/payload-pre-processing/0.log" Jul 30 11:47:52.840024 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:52.839989 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-w2xrv_baba2139-d918-4f6f-9eab-3d14183d9659/payload-processing/0.log" Jul 30 11:47:52.960076 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:52.960049 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5984b4bcb7-nxpbf_5f89b351-afdb-4b3e-8e8d-94f663c1c65b/router/0.log" Jul 30 11:47:53.462174 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:53.462141 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-pdqjc_87b51936-8941-41c4-8356-051041617c5a/main/0.log" Jul 30 11:47:53.698562 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:47:53.698534 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fp94dk_506eba34-1d5d-4afd-8447-d1d15354a095/main/0.log" Jul 30 11:48:02.400106 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:02.400055 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-452jf_f31dbccc-5060-4cb2-9df5-279124fcd806/global-pull-secret-syncer/0.log" Jul 30 11:48:02.581141 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:02.581108 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dx77j_dd19302d-83a0-404d-923b-0ed52e8bcf51/konnectivity-agent/0.log" Jul 30 11:48:02.681422 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:02.681342 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-131.ec2.internal_e190c0b1e0328f4dc4c1440290f7e5c1/haproxy/0.log" Jul 30 11:48:07.766656 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:07.766629 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-rk8vx_bcbcd504-d10a-4db5-b254-05f7e39ed612/manager/0.log" Jul 30 11:48:07.884566 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:07.884533 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-mk2vz_ec5fe991-c2f0-4ce5-aa89-4fae2bc2c755/manager/0.log" Jul 30 11:48:08.003589 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:08.003557 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-zqmss_45a7ed45-e7c6-4faa-b60b-1f2c62584111/manager/0.log" Jul 30 11:48:09.374104 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.374069 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8/alertmanager/0.log" Jul 30 11:48:09.396762 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.396729 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8/config-reloader/0.log" Jul 30 11:48:09.418926 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.418905 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8/kube-rbac-proxy-web/0.log" Jul 30 11:48:09.440540 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.440505 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8/kube-rbac-proxy/0.log" Jul 30 11:48:09.465114 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.465077 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8/kube-rbac-proxy-metric/0.log" Jul 30 11:48:09.486944 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.486926 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8/prom-label-proxy/0.log" Jul 30 11:48:09.523141 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.523119 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0a65dba3-3f5a-4a3e-bdbb-a43724fb0cf8/init-config-reloader/0.log" Jul 30 11:48:09.589335 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.589313 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-fz54r_9a6ca007-2d7e-45d8-861e-79c9b7d07655/cluster-monitoring-operator/0.log" Jul 30 11:48:09.891375 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.891347 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-svgfs_cfdce1f4-44e6-4e54-84c1-eb7f45512086/node-exporter/0.log" Jul 30 11:48:09.910282 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.910258 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-svgfs_cfdce1f4-44e6-4e54-84c1-eb7f45512086/kube-rbac-proxy/0.log" Jul 30 11:48:09.946237 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:09.946215 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-svgfs_cfdce1f4-44e6-4e54-84c1-eb7f45512086/init-textfile/0.log" Jul 30 11:48:10.300543 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.300512 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-48hr7_cc07d297-c953-45b8-81c5-f187d601e331/prometheus-operator/0.log" Jul 30 11:48:10.320697 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.320671 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-48hr7_cc07d297-c953-45b8-81c5-f187d601e331/kube-rbac-proxy/0.log" Jul 30 11:48:10.384479 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.384413 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c4b78bbc6-jbxdn_a061586f-9efd-456e-a0ad-675b136287ec/telemeter-client/0.log" Jul 30 11:48:10.441887 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.441864 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c4b78bbc6-jbxdn_a061586f-9efd-456e-a0ad-675b136287ec/reload/0.log" Jul 30 11:48:10.500506 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.500483 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5c4b78bbc6-jbxdn_a061586f-9efd-456e-a0ad-675b136287ec/kube-rbac-proxy/0.log" Jul 30 11:48:10.579581 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.579554 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7dc8f8bb-cxqtq_581526e9-9662-47f1-97d5-7996788a978e/thanos-query/0.log" Jul 30 11:48:10.626749 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.626716 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7dc8f8bb-cxqtq_581526e9-9662-47f1-97d5-7996788a978e/kube-rbac-proxy-web/0.log" Jul 30 11:48:10.646419 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.646359 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7dc8f8bb-cxqtq_581526e9-9662-47f1-97d5-7996788a978e/kube-rbac-proxy/0.log" Jul 30 11:48:10.672665 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.672648 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7dc8f8bb-cxqtq_581526e9-9662-47f1-97d5-7996788a978e/prom-label-proxy/0.log" Jul 30 11:48:10.698336 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.698311 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7dc8f8bb-cxqtq_581526e9-9662-47f1-97d5-7996788a978e/kube-rbac-proxy-rules/0.log" Jul 30 11:48:10.721728 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:10.721702 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c7dc8f8bb-cxqtq_581526e9-9662-47f1-97d5-7996788a978e/kube-rbac-proxy-metrics/0.log" Jul 30 11:48:12.170930 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.170883 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z"] Jul 30 11:48:12.171454 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171437 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83631c3a-ce95-4120-ae91-3004c098c354" containerName="payload-pre-processing" Jul 30 11:48:12.171454 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171456 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83631c3a-ce95-4120-ae91-3004c098c354" containerName="payload-pre-processing" Jul 30 11:48:12.171570 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171469 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" Jul 30 11:48:12.171570 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171476 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" Jul 30 11:48:12.171570 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171489 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" Jul 30 11:48:12.171570 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171494 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" Jul 30 11:48:12.171570 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171502 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a8de82-429b-4d72-b70d-c41125ff11df" containerName="revoke-keys" Jul 30 11:48:12.171570 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171507 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a8de82-429b-4d72-b70d-c41125ff11df" containerName="revoke-keys" Jul 30 11:48:12.171570 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171564 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="922c1710-1d7b-475a-b72d-708e7414cd49" containerName="istio-proxy" Jul 30 11:48:12.171570 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171571 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a8de82-429b-4d72-b70d-c41125ff11df" containerName="revoke-keys" Jul 30 11:48:12.171840 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.171582 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83631c3a-ce95-4120-ae91-3004c098c354" containerName="payload-pre-processing" Jul 30 11:48:12.174902 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.174887 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.177261 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.177239 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6ms7b\"/\"kube-root-ca.crt\"" Jul 30 11:48:12.177477 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.177462 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6ms7b\"/\"default-dockercfg-4xx59\"" Jul 30 11:48:12.177558 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.177479 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6ms7b\"/\"openshift-service-ca.crt\"" Jul 30 11:48:12.186334 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.186310 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z"] Jul 30 11:48:12.236892 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.236859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-proc\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.237068 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.236921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-sys\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.237068 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.237040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-lib-modules\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.237189 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.237068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrfc\" (UniqueName: \"kubernetes.io/projected/7279ec4c-3e40-4a0c-a855-db9f237ecc78-kube-api-access-7mrfc\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.237189 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.237113 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-podres\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.338102 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.338050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-lib-modules\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.338276 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.338107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrfc\" (UniqueName: \"kubernetes.io/projected/7279ec4c-3e40-4a0c-a855-db9f237ecc78-kube-api-access-7mrfc\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.338276 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.338132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-podres\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.338276 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.338155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-proc\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.338276 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.338219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-sys\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.338276 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.338238 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-lib-modules\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.338476 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.338279 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-podres\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.338476 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.338283 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-proc\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.338476 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.338305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7279ec4c-3e40-4a0c-a855-db9f237ecc78-sys\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.358335 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.358315 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrfc\" (UniqueName: \"kubernetes.io/projected/7279ec4c-3e40-4a0c-a855-db9f237ecc78-kube-api-access-7mrfc\") pod \"perf-node-gather-daemonset-89j8z\" (UID: \"7279ec4c-3e40-4a0c-a855-db9f237ecc78\") " pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.487340 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.487261 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6ms7b\"/\"default-dockercfg-4xx59\"" Jul 30 11:48:12.494069 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.494049 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:48:12.495361 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.495343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:12.497706 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.497685 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/3.log" Jul 30 11:48:12.632278 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.632253 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z"] Jul 30 11:48:12.633182 ip-10-0-138-131 kubenswrapper[2575]: W0730 11:48:12.633151 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7279ec4c_3e40_4a0c_a855_db9f237ecc78.slice/crio-7c252305e3db53ab6011a591fce7fc2210b7e1352b010637f0e03b21252a4409 WatchSource:0}: Error finding container 7c252305e3db53ab6011a591fce7fc2210b7e1352b010637f0e03b21252a4409: Status 404 returned error can't find the container with id 7c252305e3db53ab6011a591fce7fc2210b7e1352b010637f0e03b21252a4409 Jul 30 11:48:12.880520 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.880491 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:48:12.892728 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:12.892705 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mpcdj_9b8d2abb-c386-4bca-9278-c37aff6403fa/console-operator/2.log" Jul 30 11:48:13.023280 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:13.023247 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" event={"ID":"7279ec4c-3e40-4a0c-a855-db9f237ecc78","Type":"ContainerStarted","Data":"426ee619fe87f78806bc7b59ae3999abcf758a01f210174df6d05196f825fbb9"} Jul 30 11:48:13.023452 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:13.023418 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:13.023517 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:13.023462 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" event={"ID":"7279ec4c-3e40-4a0c-a855-db9f237ecc78","Type":"ContainerStarted","Data":"7c252305e3db53ab6011a591fce7fc2210b7e1352b010637f0e03b21252a4409"} Jul 30 11:48:13.043705 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:13.043667 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" podStartSLOduration=1.043651929 podStartE2EDuration="1.043651929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 11:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 11:48:13.042878745 +0000 UTC m=+3601.015739794" watchObservedRunningTime="2026-07-30 11:48:13.043651929 +0000 UTC m=+3601.016512966" Jul 30 11:48:13.772192 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:13.772154 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-8kxd8_5acafbf4-35f9-49f6-abc9-5238956900fb/volume-data-source-validator/0.log" Jul 30 11:48:14.800648 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:14.800619 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-msgck_fc7c9219-d5ac-4629-9b4f-0dcba70f47cd/dns/0.log" Jul 30 11:48:14.822601 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:14.822571 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-msgck_fc7c9219-d5ac-4629-9b4f-0dcba70f47cd/kube-rbac-proxy/0.log" Jul 30 11:48:14.880843 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:14.880815 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f5vrl_f8307c4d-a9aa-4e53-9b75-ca7bc4080028/dns-node-resolver/0.log" Jul 30 11:48:15.526265 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:15.526237 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qwpkd_177e8448-e3d4-40b5-aa14-9889d7901b64/node-ca/0.log" Jul 30 11:48:16.635256 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:16.635216 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qtsjv_df10590c-6edd-48b3-a156-bf21353dacd8/discovery/0.log" Jul 30 11:48:16.682696 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:16.682667 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8674569cb6-nlpn8_4fe82e77-f51a-452c-9213-0e779ce306ca/kube-auth-proxy/0.log" Jul 30 11:48:16.743186 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:16.743159 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-h8q2d_2d7944ed-a849-430c-b6eb-bb8bd57754c0/payload-pre-processing/0.log" Jul 30 11:48:16.773611 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:16.773584 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-w2xrv_baba2139-d918-4f6f-9eab-3d14183d9659/payload-processing/0.log" Jul 30 11:48:16.794658 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:16.794636 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5984b4bcb7-nxpbf_5f89b351-afdb-4b3e-8e8d-94f663c1c65b/router/0.log" Jul 30 11:48:17.623723 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:17.623697 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b5vtk_5bc5e620-0946-4c46-8fa7-c27d2ee33343/serve-healthcheck-canary/0.log" Jul 30 11:48:18.187507 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:18.187480 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-h8zf7_829982dc-21f7-4ee7-accb-0182ccd29add/insights-operator/0.log" Jul 30 11:48:18.188138 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:18.188117 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-h8zf7_829982dc-21f7-4ee7-accb-0182ccd29add/insights-operator/1.log" Jul 30 11:48:18.210513 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:18.210490 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7wlw_af034d34-d609-4634-9780-261ddb318624/kube-rbac-proxy/0.log" Jul 30 11:48:18.232269 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:18.232241 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7wlw_af034d34-d609-4634-9780-261ddb318624/exporter/0.log" Jul 30 11:48:18.255622 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:18.255598 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-j7wlw_af034d34-d609-4634-9780-261ddb318624/extractor/0.log" Jul 30 11:48:19.039553 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:19.039522 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6ms7b/perf-node-gather-daemonset-89j8z" Jul 30 11:48:21.107721 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:21.107691 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d558b5f66-q6g9n_b3ec4cdd-0fa1-4009-914d-5f4b5e119024/manager/0.log" Jul 30 11:48:21.209394 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:21.209313 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7f7f8_5241e5b6-69c3-42e2-b691-3ead22d1e54a/manager/0.log" Jul 30 11:48:21.406575 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:21.406547 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d97895fdd-r8wjr_cebd5340-08de-4b45-aae8-01a1ee99ab76/manager/0.log" Jul 30 11:48:21.427594 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:21.427567 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d97895fdd-r8wjr_cebd5340-08de-4b45-aae8-01a1ee99ab76/copy-manifests/0.log" Jul 30 11:48:22.842243 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:22.842215 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qtsjv_df10590c-6edd-48b3-a156-bf21353dacd8/discovery/0.log" Jul 30 11:48:22.869725 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:22.869701 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8674569cb6-nlpn8_4fe82e77-f51a-452c-9213-0e779ce306ca/kube-auth-proxy/0.log" Jul 30 11:48:22.911322 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:22.911302 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-h8q2d_2d7944ed-a849-430c-b6eb-bb8bd57754c0/payload-pre-processing/0.log" Jul 30 11:48:22.930799 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:22.930775 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-w2xrv_baba2139-d918-4f6f-9eab-3d14183d9659/payload-processing/0.log" Jul 30 11:48:22.944435 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:22.944417 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5984b4bcb7-nxpbf_5f89b351-afdb-4b3e-8e8d-94f663c1c65b/router/0.log" Jul 30 11:48:24.398104 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:24.398060 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-qtsjv_df10590c-6edd-48b3-a156-bf21353dacd8/discovery/0.log" Jul 30 11:48:24.434501 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:24.434467 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8674569cb6-nlpn8_4fe82e77-f51a-452c-9213-0e779ce306ca/kube-auth-proxy/0.log" Jul 30 11:48:24.492698 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:24.492673 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-h8q2d_2d7944ed-a849-430c-b6eb-bb8bd57754c0/payload-pre-processing/0.log" Jul 30 11:48:24.522233 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:24.522208 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-w2xrv_baba2139-d918-4f6f-9eab-3d14183d9659/payload-processing/0.log" Jul 30 11:48:24.542536 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:24.542512 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5984b4bcb7-nxpbf_5f89b351-afdb-4b3e-8e8d-94f663c1c65b/router/0.log" Jul 30 11:48:27.896727 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:27.896690 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx_7d5c371a-680e-4c9b-aac7-aa5c13018096/kube-storage-version-migrator-operator/1.log" Jul 30 11:48:27.897558 ip-10-0-138-131 kubenswrapper[2575]: I0730 11:48:27.897541 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-nwfsx_7d5c371a-680e-4c9b-aac7-aa5c13018096/kube-storage-version-migrator-operator/0.log"